Scalable topological summary construction using landmark point selection

ABSTRACT

An example method comprises receiving data points, determining at least one size of a plurality of subsets based on a constraint of at least one computation device or an analysis server, transferring each of the subsets to different computation devices, each computation device selecting a group of data points to generate a first sub-subset of landmarks, add non-landmark data points that have the farthest distance to the closest landmark to create an expanded sub-subset of landmarks, create an analysis landmark set based on a combination of expanded sub-subsets of expanded landmarks from different computation devices, perform a similarity function on the analysis landmark set, generate a cover of the mathematical reference space to create overlapping subsets, cluster the mapped landmark points based on the overlapping subsets, create a plurality of nodes, each node being based on the clustering, each landmark point being a member of at least one node.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/147,821, filed May 5, 2016, entitled “Scalable Topological SummaryConstruction Using Landmark Point Selection,” which claims the benefitof U.S. Patent Application Ser. No. 62/157,405, filed May 5, 2015,entitled “Scalable Topological Summary Construction,” and is acontinuation-in-part of and claims priority to U.S. patent applicationSer. No. 14/884,511, filed Oct. 15, 2015, entitled “Landmark PointSelection,” which is a continuation-in-part of and claims priority toU.S. patent application Ser. No. 14/639,954, filed Mar. 5, 2015,entitled “Systems and Methods for Capture of Relationships WithinInformation,” which claims priority to U.S. Provisional PatentApplication Ser. No. 61/948,490, filed Mar. 5, 2014, entitled “Systemsand Methods For Landmarked Stochastic Neighbor Embedding,” all of whichare incorporated by reference herein.

BACKGROUND 1. Field of the Invention(s)

Embodiments discussed herein are directed to grouping of data points fordata analysis and more particularly to generating a graph utilizingimproved groupings of data points based on scores of the groupings.

2. Related Art

As the collection and storage data has increased, there is an increasedneed to analyze and make sense of large amounts of data. Examples oflarge datasets may be found in financial services companies, oilexpiration, biotech, and academia. Unfortunately, previous methods ofanalysis of large multidimensional datasets tend to be insufficient (ifpossible at all) to identify important relationships and may becomputationally inefficient.

In order to process large datasets, some previous methods of analysisuse clustering. Clustering often breaks important relationships and isoften too blunt an instrument to assist in the identification ofimportant relationships in the data. Similarly, previous methods oflinear regression, projection pursuit, principal component analysis, andmultidimensional scaling often do not reveal important relationships.Further, existing linear algebraic and analytic methods are toosensitive to large scale distances and, as a result, lose detail.

Even if the data is analyzed, sophisticated experts are often necessaryto interpret and understand the output of previous methods. Althoughsome previous methods allow graphs that depict some relationships in thedata, the graphs are not interactive and require considerable time for ateam of such experts to understand the relationships. Further, theoutput of previous methods does not allow for exploratory data analysiswhere the analysis can be quickly modified to discover newrelationships. Rather, previous methods require the formulation of ahypothesis before testing.

SUMMARY OF THE INVENTION(S)

An example method comprises receiving a large number of data points,determining at least one size of a plurality of subsets of the largenumber of data points based on constraints of at least one of aplurality of computation devices or an analysis server, each data pointof the large number of data points being a member of at least one of theplurality of subsets of the large number of data points, transferringeach of the plurality of subsets of large number of data points to arespective one of the plurality of computation devices, for each of theplurality of subsets of data points by an associated computation deviceof the plurality of computation devices: selecting, by the associatedcomputation device, a group of data points from the subset of datapoints to generate a first sub-subset of landmarks, adding, by theassociated computation device, a non-landmark data point of the subsetof data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, adding the non-landmark data pointscomprising calculating first data point distances between eachnon-landmark data point and each landmark, identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point, identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapath distances, and adding the particular non-landmark data point to thefirst sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks, until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points, creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks, performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space, generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets, clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space, creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node, andconnecting at least two of the plurality of nodes with an edge if the atleast two of the plurality of nodes share at least one landmark point ofthe analysis landmark set as a member.

In some embodiments, the method further comprises for each data pointthat is both a member of the large data set but is not a member of theanalysis landmark set: determining a distance between that data pointand all landmark points of the analysis landmark set, identifying aclosest landmark of the analysis landmark set to that data point,identifying node that includes the closest landmark of the analysislandmark set, and adding that data point as a member of the node thatincludes the closest landmark of the analysis landmark set. In someembodiments, the method further comprises generating a visualization ofthe plurality of nodes and edge.

In various embodiments, the method further comprises for each data pointthat is both a member of the large data set but is not a member of theanalysis landmark set: determining a distance between that data pointand all landmark points of the analysis landmark set, identifying aclosest landmark of the analysis landmark set to that data point,comparing a distance between the closest landmark of the analysislandmark set and that data point to a node threshold, and if thedistance between the closest landmark of the analysis landmark set andthat data point is greater than the node threshold, generating a newnode including that data point as a member of the new node, if thedistance the distance between the closest landmark of the analysislandmark set and that data point is less than the node threshold, addingthat data point as a member of the node that includes the closestlandmark of the analysis landmark set.

The method may further comprise for each data point that is both amember of the large data set but is not a member of the analysislandmark set: determining a distance between that data point and alllandmark points of the analysis landmark set, identifying apredetermined number of closest landmark of the analysis landmark set tothat data point, identifying a node which includes a majority of thepredetermined number of closest landmarks of the analysis landmark setas members, and adding that data point as a member of the node thatincludes a majority of the predetermined number of closest landmarks ofthe analysis landmark set as members. The method may further comprisegenerating a visualization of the plurality of nodes and edge.

In various embodiments, the method comprises determining thepredetermined number of members of the expanded sub-subset of theexpanded landmarks based on the constraints of the at least one of aplurality of computation devices or an analysis server. The method mayalso further comprise wherein the determination of the predeterminednumber of members of the expanded sub-subset of the expanded landmarksis based, at least in part, on a determination of a predetermined numberof members of the analysis landmark set. Selecting, by the associatedcomputation device, the group of data points from the subset of datapoints to generate the first sub-subset of landmarks may be performedrandomly.

An example non-transitory computer readable medium may compriseinstructions executable by a processor to perform a method, the methodcomprising: receiving a large number of data points, determining atleast one size of a plurality of subsets of the large number of datapoints based on constraints of at least one of a plurality ofcomputation devices or an analysis server, each data point of the largenumber of data points being a member of at least one of the plurality ofsubsets of the large number of data points, transferring each of theplurality of subsets of large number of data points to a respective oneof the plurality of computation devices, each of the plurality ofsubsets of data points by an associated computation device of theplurality of computation devices being configured to: select, by theassociated computation device, a group of data points from the subset ofdata points to generate a first sub-subset of landmarks, add, by theassociated computation device, a non-landmark data point of the subsetof data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, add the non-landmark data points, theadding the non-landmark data points comprising calculating first datapoint distances between each non-landmark data point and each landmark,identifying a shortest data point distance from among the first datapoint distances for each non-landmark data point, identifying aparticular non-landmark data point with a longest first landmarkdistance of all the shortest data path distances, and adding theparticular non-landmark data point to the first sub-subset of landmarksto expand the first sub-subset of landmarks to generate an expanded setof landmarks, until the expanded sub-subset of the expanded landmarksreaches a predetermined number of members, repeat adding thenon-landmark data points, creating an analysis landmark set based on acombination of expanded sub-subsets of expanded landmarks, performing asimilarity function on the analysis landmark set to map landmark pointsof the analysis landmark set to a mathematical reference space,generating a cover of the mathematical reference space to divide themathematical reference space into overlapping subsets, clustering themapped landmark points of the analysis landmark set based on theoverlapping subsets of the cover in the mathematical reference space,creating a plurality of nodes, each of the plurality of nodes beingbased on the clustering of the mapped landmark points of the analysislandmark set, each landmark point of the analysis landmark set being amember of at least one node, and connecting at least two of theplurality of nodes with an edge if the at least two of the plurality ofnodes share at least one landmark point of the analysis landmark set asa member.

An example system includes memory and a processor, the memory mayinclude instructions to configure the processor to receive a largenumber of data points, determine at least one size of a plurality ofsubsets of the large number of data points based on constraints of atleast one of a plurality of computation devices or an analysis server,each data point of the large number of data points being a member of atleast one of the plurality of subsets of the large number of datapoints, transfer each of the plurality of subsets of large number ofdata points to a respective one of the plurality of computation devicesto enable for each of the plurality of subsets of data points by anassociated computation device of the plurality of computation devices:select, by the associated computation device, a group of data pointsfrom the subset of data points to generate a first sub-subset oflandmarks, add, by the associated computation device, a non-landmarkdata point of the subset of data points to the first sub-subset oflandmarks to create an expanded sub-subset of landmarks, adding thenon-landmark data points comprising: calculate first data pointdistances between each non-landmark data point and each landmark,identify a shortest data point distance from among the first data pointdistances for each non-landmark data point, identify a particularnon-landmark data point with a longest first landmark distance of allthe shortest data path distances, and add the particular non-landmarkdata point to the first sub-subset of landmarks to expand the firstsub-subset of landmarks to generate an expanded set of landmarks untilthe expanded sub-subset of the expanded landmarks reaches apredetermined number of members, repeat adding the non-landmark datapoints, create an analysis landmark set based on a combination ofexpanded sub-subsets of expanded landmarks, perform a similarityfunction on the analysis landmark set to map landmark points of theanalysis landmark set to a mathematical reference space, generate acover of the mathematical reference space to divide the mathematicalreference space into overlapping subsets, cluster the mapped landmarkpoints of the analysis landmark set based on the overlapping subsets ofthe cover in the mathematical reference space, create a plurality ofnodes, each of the plurality of nodes being based on the clustering ofthe mapped landmark points of the analysis landmark set, each landmarkpoint of the analysis landmark set being a member of at least one node,and connect at least two of the plurality of nodes with an edge if theat least two of the plurality of nodes share at least one landmark pointof the analysis landmark set as a member.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an example graph representing data that appears to be dividedinto three disconnected groups.

FIG. 1B is an example graph representing data set obtained from aLotka-Volterra equation modeling the populations of predators and preyover time.

FIG. 1C is an example graph of data sets whereby the data does not breakup into disconnected groups, but instead has a structure in which thereare lines (or flares) emanating from a central group.

FIG. 2 is an example environment in which embodiments may be practiced.

FIG. 3 is a block diagram of an example analysis server.

FIG. 4 is a flow chart depicting an example method of dataset analysisand visualization in some embodiments.

FIG. 5 is an example ID field selection interface window in someembodiments.

FIG. 6a is an example data field selection interface window in someembodiments.

FIG. 6b is an example metric and filter selection interface window insome embodiments.

FIG. 7 is an example filter parameter interface window in someembodiments.

FIG. 8 is a flowchart for data analysis and generating a visualizationin some embodiments.

FIG. 9 is an example interactive visualization in some embodiments.

FIG. 10 is an example interactive visualization displaying an explaininformation window in some embodiments.

FIG. 11 is a flowchart of functionality of the interactive visualizationin some embodiments.

FIG. 12 is a flowchart of for generating a cancer map visualizationutilizing biological data of a plurality of patients in someembodiments.

FIG. 13 is an example data structure including biological data for anumber of patients that may be used to generate the cancer mapvisualization in some embodiments.

FIG. 14 is an example visualization displaying the cancer map in someembodiments.

FIG. 15 is a flowchart of for positioning new patient data relative tothe cancer map visualization in some embodiments.

FIG. 16 is an example visualization displaying the cancer map includingpositions for three new cancer patients in some embodiments.

FIG. 17 is a flowchart of utilization the visualization and positioningof new patient data in some embodiments

FIG. 18 is an example digital device in some embodiments.

FIG. 19 shows an example landmark module configured to identify landmarkpoints that approximate or represent a larger collection of data pointsin accordance with various embodiments.

FIG. 20 is a flowchart for generating a set of landmark points in someembodiments.

FIG. 21A shows example metric space containing data in accordance withvarious embodiments.

FIG. 21B shows subset composed of individual data points in accordancewith some embodiments.

FIG. 21C shows exemple random landmarks R₁, R₂, R₃, and R₄ that havebeen randomly selected as initial landmarks in the subset identified inFIG. 21A.

FIG. 21D shows lines corresponding to data point distances to eachlandmark for three points (P₁, P₂, and P₃) in the subset identified inFIG. 21A.

FIG. 22A shows example data point distances between point P₁ and randomlandmarks R₁, R₂, R₃, and R₄.

FIG. 22B shows example distances between point P₂ and random landmarksR₁, R₂, R₃, and R₄.

FIG. 22C shows an example table wherein distances for each point arestored.

FIG. 23A shows example landmark distances for points P₁, P₂, and P₃ tolandmark R₁ which can be used to demonstrate the selection of additionallandmark points.

FIG. 23B shows example shortest distances from each non-landmark pointto each landmark point.

FIG. 23C shows point P₂ as new MM landmark point L₁ in this example.

FIG. 23D shows subset with L₁ as a new landmark where the distancesbetween various points have been calculated.

FIG. 24A shows an example wherein data in X does not fit into localmemory (e.g., Random Access Memory (RAM)) and is, therefore, read off oflong term storage.

FIG. 24B shows an example wherein data point sets are stored in localmemor instead of the landmark set in accordance with variousembodiments.

FIG. 25A shows subset with distances shown for points P₁, P₂, P₃, and P₄to their respective closest random landmark (R₁, R₂, R₃, R₄).

FIG. 25B shows example shortest distances from each non-landmark pointto each landmark point.

FIG. 25C shows points P₂, P₃, and P₄ as landmarks L₁, L₂, and L₃ in thisexample.

FIG. 26 is an example environment in which embodiments may be practiced.

FIG. 27 is a flowchart for determining landmark points using any numberof computation devices in some embodiments.

FIG. 28 is a flowchart for a computation device to create an expandedlandmark subset of landmark points in some embodiments.

FIG. 29 is a flowchart for a method of the analysis system to identifynodes (e.g., graphical nodes or vertices) associated with one or more ofthe landmark points of the analysis landmark set of landmark points.

FIG. 30 is a flowchart for adding non-landmark points as members ofnodes in some embodiments.

FIG. 31 is another flowchart for adding non-landmark points as membersof nodes in some embodiments.

FIG. 32 is a flowchart for adding non-landmark points as members ofnodes that share one or more characteristic(s) in some embodiments.

DETAILED DESCRIPTION OF DRAWINGS

Some embodiments described herein may be a part of the subject ofTopological Data Analysis (TDA). TDA is an area of research which hasproduced methods for studying point cloud data sets from a geometricpoint of view. Other data analysis techniques use “approximation bymodels” of various types. Examples of other data analysis techniquesinclude regression methods which model data as a graph of a function inone or more variables. Unfortunately, certain qualitative properties(which one can readily observe when the data is two-dimensional) may beof a great deal of importance for understanding, and these features maynot be readily represented within such models.

FIG. 1A is an example graph representing data that appears to be dividedinto three disconnected groups. In this example, the data for this graphmay be associated with various physical characteristics related todifferent population groups or biomedical data related to differentforms of a disease. Seeing that the data breaks into groups in thisfashion can give insight into the data, once one understands whatcharacterizes the groups.

FIG. 1B is an example graph representing data set obtained from aLotka-Volterra equation modeling the populations of predators and preyover time. From FIG. 1B, one observation about this data is that it isarranged in a loop. The loop is not exactly circular, but it istopologically a circle. The exact form of the equations, whileinteresting, may not be of as much importance as this qualitativeobservation which reflects the fact that the underlying phenomenon isrecurrent or periodic. When looking for periodic or recurrent phenomena,methods may be developed which can detect the presence of loops withoutdefining explicit models. For example, periodicity may be detectablewithout having to first develop a fully accurate model of the dynamics.

FIG. 1C is an example graph of data sets whereby the data does not breakup into disconnected groups, but instead has a structure in which thereare lines (or flares) emanating from a central group. In this case, thedata also suggests the presence of three distinct groups, but theconnectedness of the data does not reflect this. This particular datathat is the basis for the example graph in FIG. 1C arises from a studyof single nucleotide polymorphisms (SNPs).

In each of the examples above, aspects of the shape of the data arerelevant in reflecting information about the data. Connectedness (thesimplest property of shape) reflects the presence of a discreteclassification of the data into disparate groups. The presence of loops,another simple aspect of shape, often reflect periodic or recurrentbehavior. Finally, in the third example, the shape containing flaressuggests a classification of the data descriptive of ways in whichphenomena can deviate from the norm, which would typically berepresented by the central core. These examples support the idea thatthe shape of data (suitably defined) is an important aspect of itsstructure, and that it is therefore important to develop methods foranalyzing and understanding its shape. The part of mathematics whichconcerns itself with the study of shape is called topology, andtopological data analysis attempts to adapt methods for studying shapewhich have been developed in pure mathematics to the study of the shapeof data, suitably defined.

One question is how notions of geometry or shape are translated intoinformation about point clouds, which are, after all, finite sets? Whatwe mean by shape or geometry can come from a dissimilarity function ormetric (e.g., a non-negative, symmetric, real-valued function d on theset of pairs of points in the data set which may also satisfy thetriangle inequality, and d(x; y)=0 if and only if x=y). Such functionsexist in profusion for many data sets. For example, when data comes inthe form of a numerical matrix, where the rows correspond to the datapoints and the columns are the fields describing the data, then-dimensional Euclidean distance function is natural when there are nfields. Similarly, in this example, there are Pearson correlationdistances, cosine distances, and other choices.

When the data is not Euclidean, for example if one is consideringgenomic sequences, various notions of distance may be defined usingmeasures of similarity based on Basic Local Alignment Search Tool(BLAST) type similarity scores. Further, a measure of similarity cancome in non-numeric forms, such as social networks of friends orsimilarities of hobbies, buying patterns, tweeting, and/or professionalinterests. In any of these ways the notion of shape may be formulatedvia the establishment of a useful notion of similarity of data points.

One of the advantages of TDA is that TDA may depend on nothing more thansuch a notion, which is a very primitive or low-level model. TDA mayrely on many fewer assumptions than standard linear or algebraic models,for example. Further, the methodology may provide new ways ofvisualizing and compressing data sets, which facilitate understandingand monitoring data. The methodology may enable study ofinterrelationships among disparate data sets and/ormultiscale/multiresolution study of data sets. Moreover, the methodologymay enable interactivity in the analysis of data, using point and clickmethods.

In some embodiments, TDA may be a very useful complement to moretraditional methods, such as Principal Component Analysis (PCA),multidimensional scaling, and hierarchical clustering. These existingmethods are often quite useful, but suffer from significant limitations.PCA, for example, is an essentially linear procedure and there aretherefore limits to its utility in highly non-linear situations.Multidimensional scaling is a method which is not intrinsically linear,but can in many situations wash out detail, since it may overweightlarge distances. In addition, when metrics do not satisfy an intrinsicflatness condition, it may have difficulty in faithfully representingthe data. Hierarchical clustering does exhibit multiscale behavior, butrepresents data only as disjoint clusters, rather than retaining any ofthe geometry of the data set. In all four cases, these limitationsmatter for many varied kinds of data.

We now summarize example properties of an example construction, in someembodiments, which may be used for representing the shape of data setsin a useful, understandable fashion as a finite graph:

-   -   The input may be a collection of data points equipped in some        way with a distance or dissimilarity function, or other        description. This can be given implicitly when the data is in        the form of a matrix, or explicitly as a matrix of distances or        even the generating edges of a mathematical network.    -   One construction may also use one or more lens functions (i.e.        real valued functions on the data). Lens function(s) may depend        directly on the metric. For example, lens function(s) might be        the result of a density estimator or a measure of centrality or        data depth. Lens function(s) may, in some embodiments, depend on        a particular representation of the data, as when one uses the        first one or two coordinates of a principal component or        multidimensional scaling analysis. In some embodiments, the lens        function(s) may be columns which expert knowledge identifies as        being intrinsically interesting, as in cholesterol levels and        BMI in a study of heart disease.    -   In some embodiments, the construction may depend on a choice of        two or more processing parameters, resolution, and gain.        Increase in resolution typically results in more nodes and an        increase in the gain increases the number of edges in a        visualization and/or graph in a reference space as further        described herein.    -   The output may be, for example, a visualization (e.g., a display        of connected nodes or “network”) or simplicial complex. One        specific combinatorial formulation in one embodiment may be that        the vertices form a finite set, and then the additional        structure may be a collection of edges (unordered pairs of        vertices) which are pictured as connections in this network.

In various embodiments, a system for handling, analyzing, andvisualizing data using drag and drop methods as opposed to text basedmethods is described herein. Philosophically, data analytic tools arenot necessarily regarded as “solvers,” but rather as tools forinteracting with data. For example, data analysis may consist of severaliterations of a process in which computational tools point to regions ofinterest in a data set. The data set may then be examined by people withdomain expertise concerning the data, and the data set may then besubjected to further computational analysis. In some embodiments,methods described herein provide for going back and forth betweenmathematical constructs, including interactive visualizations (e.g.,graphs), on the one hand and data on the other.

In one example of data analysis in some embodiments described herein, anexemplary clustering tool is discussed which may be more powerful thanexisting technology, in that one can find structure within clusters andstudy how clusters change over a period of time or over a change ofscale or resolution.

An example interactive visualization tool (e.g., a visualization modulewhich is further described herein) may produce combinatorial output inthe form of a graph which can be readily visualized. In someembodiments, the example interactive visualization tool may be lesssensitive to changes in notions of distance than current methods, suchas multidimensional scaling.

Some embodiments described herein permit manipulation of the data from avisualization. For example, portions of the data which are deemed to beinteresting from the visualization can be selected and converted intodatabase objects, which can then be further analyzed. Some embodimentsdescribed herein permit the location of data points of interest withinthe visualization, so that the connection between a given visualizationand the information the visualization represents may be readilyunderstood.

FIG. 2 is an example environment 200 in which embodiments may bepracticed. In various embodiments, data analysis and interactivevisualization may be performed locally (e.g., with software and/orhardware on a local digital device), across a network (e.g., via cloudcomputing), or a combination of both. In many of these embodiments, adata structure is accessed to obtain the data for the analysis, theanalysis is performed based on properties and parameters selected by auser, and an interactive visualization is generated and displayed. Thereare many advantages between performing all or some activities locallyand many advantages of performing all or some activities over a network.

Environment 200 comprises user devices 200 a-202 n, a communicationnetwork 204, data storage server 206, and analysis server 208.Environment 200 depicts an embodiment wherein functions are performedacross a network. In this example, the user(s) may take advantage ofcloud computing by storing data in a data storage server 206 over acommunication network 204. The analysis server 208 may perform analysisand generation of an interactive visualization.

User devices 202 a-202 n may be any digital devices. A digital device isany device that includes memory and a processor. Digital devices arefurther described in FIG. 18. The user devices 202 a-202 n may be anykind of digital device that may be used to access, analyze and/or viewdata including, but not limited to a desktop computer, laptop, notebook,or other computing device.

In various embodiments, a user, such as a data analyst, may generateand/or receive a database or other data structure with the user device202 a to be saved to the data storage server 206. The user device 202 amay communicate with the analysis server 208 via the communicationnetwork 204 to perform analysis, examination, and visualization of datawithin the database.

The user device 202 a may comprise any number of client programs. One ormore of the client programs may interact with one or more applicationson the analysis server 208. In other embodiments, the user device 202 amay communicate with the analysis server 208 using a browser or otherstandard program. In various embodiments, the user device 202 acommunicates with the analysis server 208 via a virtual private network.Those skilled in the art will appreciate that that communication betweenthe user device 202 a, the data storage server 206, and/or the analysisserver 208 may be encrypted or otherwise secured.

The communication network 204 may be any network that allows digitaldevices to communicate. The communication network 204 may be theInternet and/or include LAN and WANs. The communication network 204 maysupport wireless and/or wired communication.

The data storage server 206 is a digital device that is configured tostore data. In various embodiments, the data storage server 206 storesdatabases and/or other data structures. The data storage server 206 maybe a single server or a combination of servers. In one example the datastorage server 206 may be a secure server wherein a user may store dataover a secured connection (e.g., via https). The data may be encryptedand backed-up. In some embodiments, the data storage server 206 isoperated by a third-party such as Amazon's S3 service.

The database or other data structure may comprise large high-dimensionaldatasets. These datasets are traditionally very difficult to analyzeand, as a result, relationships within the data may not be identifiableusing previous methods. Further, previous methods may be computationallyinefficient.

The analysis server 208 may include any number of digital devicesconfigured to analyze data (e.g., the data in the stored database and/orother dataset received and/or generated by the user device 202 a).Although only one digital device is depicted in FIG. 2 corresponding tothe analysis server 208, it will be appreciated that any number offunctions of the analysis server 208 may be performed by any number ofdigital devices.

In various embodiments, the analysis server 208 may perform manyfunctions to interpret, examine, analyze, and display data and/orrelationships within data. In some embodiments, the analysis server 208performs, at least in part, topological analysis of large datasetsapplying metrics, filters, and resolution parameters chosen by the user.The analysis is further discussed regarding FIG. 8 herein.

The analysis server 208 may generate graphs in memory, visualizedgraphs, and/or an interactive visualization of the output of theanalysis. The interactive visualization allows the user to observe andexplore relationships in the data. In various embodiments, theinteractive visualization allows the user to select nodes comprisingdata that has been clustered. The user may then access the underlyingdata, perform further analysis (e.g., statistical analysis) on theunderlying data, and manually reorient the graph(s) (e.g., structures ofnodes and edges described herein) within the interactive visualization.The analysis server 208 may also allow for the user to interact with thedata, see the graphic result. The interactive visualization is furtherdiscussed in FIGS. 9-11.

The graphs in memory and/or visualized graphs may also include nodesand/or edges as described herein. Graphs that are generated in memorymay not be depicted to a user but rather may be in memory of a digitaldevice. Visualized graphs are rendered graphs that may be depicted tothe user (e.g., using user device 202 a).

In some embodiments, the analysis server 208 interacts with the userdevice(s) 202 a-202 n over a private and/or secure communicationnetwork. The user device 202 a may include a client program that allowsthe user to interact with the data storage server 206, the analysisserver 208, another user device (e.g., user device 202 n), a database,and/or an analysis application executed on the analysis server 208.

It will be appreciated that all or part of the data analysis may occurat the user device 202 a. Further, all or part of the interaction withthe visualization (e.g., graphic) may be performed on the user device202 a. Alternately, all or part of the data analysis may occur on anynumber of digital devices including, for example, on the analysis server208.

Although two user devices 202 a and 202 n are depicted, those skilled inthe art will appreciate that there may be any number of user devices inany location (e.g., remote from each other). Similarly, there may be anynumber of communication networks, data storage servers, and analysisservers.

Cloud computing may allow for greater access to large datasets (e.g.,via a commercial storage service) over a faster connection. Further,those skilled in the art will appreciate that services and computingresources offered to the user(s) may be scalable.

FIG. 3 is a block diagram of an example analysis server 208. In someembodiments, the analysis server 208 comprises a processor 302,input/output (I/O) interface 304, a communication network interface 306,a memory system 308, a storage system 310, and a processing module 312.The processor 302 may comprise any processor or combination ofprocessors with one or more cores.

The input/output (I/O) interface 304 may comprise interfaces for variousI/O devices such as, for example, a keyboard, mouse, and display device.The example communication network interface 306 is configured to allowthe analysis server 208 to communication with the communication network204 (see FIG. 2). The communication network interface 306 may supportcommunication over an Ethernet connection, a serial connection, aparallel connection, and/or an ATA connection. The communication networkinterface 306 may also support wireless communication (e.g.,802.11a/b/g/n, WiMax, LTE, WiFi). It will be apparent to those skilledin the art that the communication network interface 306 can support manywired and wireless standards.

The memory system 308 may be any kind of memory including RAM, ROM, orflash, cache, virtual memory, etc. In various embodiments, working datais stored within the memory system 308. The data within the memorysystem 308 may be cleared or ultimately transferred to the storagesystem 310.

The storage system 310 includes any storage configured to retrieve andstore data. Some examples of the storage system 310 include flashdrives, hard drives, optical drives, and/or magnetic tape. Each of thememory system 308 and the storage system 310 comprises a non-transitorycomputer-readable medium, which stores instructions (e.g., softwareprograms) executable by processor 302.

The storage system 310 comprises a plurality of modules utilized byembodiments of discussed herein. A module may be hardware, software(e.g., including instructions executable by a processor), or acombination of both. In one embodiment, the storage system 310 includesa processing module 312. The processing module 312 may include memoryand/or hardware and includes an input module 314, a filter module 316, aresolution module 318, an analysis module 320, a visualization engine322, and database storage 324. Alternative embodiments of the analysisserver 208 and/or the storage system 310 may comprise more, less, orfunctionally equivalent components and modules.

The input module 314 may be configured to receive commands andpreferences from the user device 202 a. In various examples, the inputmodule 314 receives selections from the user which will be used toperform the analysis. The output of the analysis may be an interactivevisualization.

The input module 314 may provide the user a variety of interface windowsallowing the user to select and access a database, choose fieldsassociated with the database, choose a metric, choose one or morefilters, and identify resolution parameters for the analysis. In oneexample, the input module 314 receives a database identifier andaccesses a large multidimensional database. The input module 314 mayscan the database and provide the user with an interface window allowingthe user to identify an ID field. An ID field is an identifier for eachdata point. In one example, the identifier is unique. The same columnname may be present in the table from which filters are selected. Afterthe ID field is selected, the input module 314 may then provide the userwith another interface window to allow the user to choose one or moredata fields from a table of the database.

Although interactive windows may be described herein, those skilled inthe art will appreciate that any window, graphical user interface,and/or command line may be used to receive or prompt a user or userdevice 202 a for information.

The filter module 316 may subsequently provide the user with aninterface window to allow the user to select a metric to be used inanalysis of the data within the chosen data fields. The filter module316 may also allow the user to select and/or define one or more filters.

The resolution module 318 may allow the user to select a resolution,including filter parameters. In one example, the user enters a number ofintervals and a percentage overlap for a filter.

The analysis module 320 may perform data analysis based on the databaseand the information provided by the user. In various embodiments, theanalysis module 320 performs an algebraic topological analysis toidentify structures and relationships within data and clusters of data.Those skilled in the art will appreciate that the analysis module 320may use parallel algorithms or use generalizations of variousstatistical techniques (e.g., generalizing the bootstrap to zig-zagmethods) to increase the size of data sets that can be processed. Theanalysis is further discussed herein (e.g., see discussion regardingFIG. 8). It will be appreciated that the analysis module 320 is notlimited to algebraic topological analysis but may perform any analysis.

The visualization engine 322 generates an interactive visualizationbased on the output from the analysis module 320. The interactivevisualization allows the user to see all or part of the analysisgraphically. The interactive visualization also allows the user tointeract with the visualization. For example, the user may selectportions of a graph from within the visualization to see and/or interactwith the underlying data and/or underlying analysis. The user may thenchange the parameters of the analysis (e.g., change the metric,filter(s), or resolution(s)) which allows the user to visually identifyrelationships in the data that may be otherwise undetectable using priormeans. The interactive visualization is further described herein (e.g.,see discussion regarding FIGS. 9-11).

The database storage 324 is configured to store all or part of thedatabase that is being accessed. In some embodiments, the databasestorage 324 may store saved portions of the database. Further, thedatabase storage 324 may be used to store user preferences, parameters,and analysis output thereby allowing the user to perform many differentfunctions on the database without losing previous work.

Those skilled in the art will appreciate that that all or part of theprocessing module 312 may be at the user device 202 a or the databasestorage server 206. In some embodiments, all or some of thefunctionality of the processing module 312 may be performed by the userdevice 202 a.

In various embodiments, systems and methods discussed herein may beimplemented with one or more digital devices. In some examples, someembodiments discussed herein may be implemented by a computer program(instructions) executed by a processor. The computer program may providea graphical user interface. Although such a computer program isdiscussed, those skilled in the art will appreciate that embodiments maybe performed using any of the following, either alone or in combination,including, but not limited to, a computer program, multiple computerprograms, firmware, and/or hardware.

A module and/or engine may include any processor or combination ofprocessors. In some examples, a module and/or engine may include or be apart of a processor, digital signal processor (DSP), applicationspecific integrated circuit (ASIC), an integrated circuit, and/or thelike. In various embodiments, the module and/or engine may be softwareor firmware.

FIG. 4 is a flow chart 400 depicting an example method of datasetanalysis and visualization in some embodiments. In step 402, the inputmodule 314 accesses a database. The database may be any data structurecontaining data (e.g., a very large dataset of multidimensional data).In some embodiments, the database may be a relational database. In someexamples, the relational database may be used with MySQL, Oracle,Microsoft SQL Server, Aster nCluster, Teradata, and/or Vertica. Thoseskilled in the art will appreciate that the database may not be arelational database.

In some embodiments, the input module 314 receives a database identifierand a location of the database (e.g., the data storage server 206) fromthe user device 202 a (see FIG. 2). The input module 314 may then accessthe identified database. In various embodiments, the input module 314may read data from many different sources, including, but not limited toMS Excel files, text files (e.g., delimited or CSV), Matlab .mat format,or any other file.

In some embodiments, the input module 314 receives an IP address orhostname of a server hosting the database, a username, password, and thedatabase identifier. This information (herein referred to as “connectioninformation”) may be cached for later use. It will be appreciated thatthe database may be locally accessed and that all, some, or none of theconnection information may be required. In one example, the user device202 a may have full access to the database stored locally on the userdevice 202 a so the IP address is unnecessary. In another example, theuser device 202 a may already have loaded the database and the inputmodule 314 merely begins by accessing the loaded database.

In various embodiments, the identified database stores data withintables. A table may have a “column specification” which stores the namesof the columns and their data types. A “row” in a table, may be a tuplewith one entry for each column of the correct type. In one example, atable to store employee records might have a column specification suchas:

-   -   employee_id primary key int (this may store the employee's ID as        an integer, and uniquely identifies a row)    -   age int    -   gender char(1) (gender of the employee may be a single character        either M or F)    -   salary double (salary of an employee may be a floating point        number)    -   name varchar (name of the employee may be a variable-length        string)        In this example, each employee corresponds to a row in this        table. Further, the tables in this example relational database        are organized into logical units called databases. An analogy to        file systems is that databases can be thought of as folders and        files as tables. Access to databases may be controlled by the        database administrator by assigning a username/password pair to        authenticate users.

Once the database is accessed, the input module 314 may allow the userto access a previously stored analysis or to begin a new analysis. Ifthe user begins a new analysis, the input module 314 may provide theuser device 202 a with an interface window allowing the user to identifya table from within the database. In one example, the input module 314provides a list of available tables from the identified database.

In step 404, the input module 314 receives a table identifieridentifying a table from within the database. The input module 314 maythen provide the user with a list of available ID fields from the tableidentifier. In step 406, the input module 314 receives the ID fieldidentifier from the user and/or user device 202 a. The ID field is, insome embodiments, the primary key.

Having selected the primary key, the input module 314 may generate a newinterface window to allow the user to select data fields for analysis.In step 408, the input module 314 receives data field identifiers fromthe user device 202 a. The data within the data fields may be lateranalyzed by the analysis module 320.

In step 408, the filter module 316 selects one or more filters. In someembodiments, the filter module 316 and/or the input module 314 generatesan interface window allowing the user of the user device 202 a optionsfor a variety of different metrics and filter preferences. The interfacewindow may be a drop down menu identifying a variety of distance metricsto be used in the analysis.

In some embodiments, the user selects and/or provides filteridentifier(s) to the filter module 316. The role of the filters in theanalysis is also further described herein. The filters, for example, maybe user defined, geometric, or based on data which has beenpre-processed. In some embodiments, the data based filters are numericalarrays which can assign a set of real numbers to each row in the tableor each point in the data generally.

A variety of geometric filters may be available for the user to choose.Geometric filters may include, but are not limited to:

-   -   Density    -   L1 Eccentricity    -   L-infinity Eccentricity    -   Witness based Density    -   Witness based Eccentricity    -   Eccentricity as distance from a fixed point    -   Approximate Kurtosis of the Eccentricity

In step 410, the filter module 316 identifies a metric. Metric optionsmay include, but are not limited to, Euclidean, DB Metric, variancenormalized Euclidean, and total normalized Euclidean. The metric and theanalysis are further described herein.

In step 412, the resolution module 318 defines the resolution to be usedwith a filter in the analysis. The resolution may comprise a number ofintervals and an overlap parameter. In various embodiments, theresolution module 318 allows the user to adjust the number of intervalsand overlap parameter (e.g., percentage overlap) for one or morefilters.

In step 414, the analysis module 320 processes data of selected fieldsbased on the metric, filter(s), and resolution(s) to generate thevisualization. This process is further discussed herein (e.g., seediscussion regarding FIG. 8).

In step 416, the visualization engine 322 displays the interactivevisualization. In various embodiments, the visualization may be renderedin two or three dimensional space. The visualization engine 322 may usean optimization algorithm for an objective function which is correlatedwith good visualization (e.g., the energy of the embedding). Thevisualization may show a collection of nodes corresponding to each ofthe partial clusters in the analysis output and edges connecting them asspecified by the output. The interactive visualization is furtherdiscussed herein (e.g., see discussion regarding FIGS. 9-11).

Although many examples discuss the input module 314 as providinginterface windows, it will be appreciated that all or some of theinterface may be provided by a client on the user device 202 a. Further,in some embodiments, the user device 202 a may be running all or some ofthe processing module 312.

FIGS. 5-7 depict various interface windows to allow the user to makeselections, enter information (e.g., fields, metrics, and filters),provide parameters (e.g., resolution), and provide data (e.g., identifythe database) to be used with analysis. It will be appreciated that anygraphical user interface or command line may be used to make selections,enter information, provide parameters, and provide data.

FIG. 5 is an exemplary ID field selection interface window 500 in someembodiments. The ID field selection interface window 500 allows the userto identify an ID field. The ID field selection interface window 500comprises a table search field 502, a table list 504, and a fieldsselection window 506.

In various embodiments, the input module 314 identifies and accesses adatabase from the database storage 324, user device 202 a, or the datastorage server 206. The input module 314 may then generate the ID fieldselection interface window 500 and provide a list of available tables ofthe selected database in the table list 504. The user may click on atable or search for a table by entering a search query (e.g., a keyword)in the table search field 502. Once a table is identified (e.g., clickedon by the user), the fields selection window 506 may provide a list ofavailable fields in the selected table. The user may then choose a fieldfrom the fields selection window 506 to be the ID field. In someembodiments, any number of fields may be chosen to be the ID field(s).

FIG. 6a is an example data field selection interface window 600 a insome embodiments. The data field selection interface window 600 a allowsthe user to identify data fields. The data field selection interfacewindow 600 a comprises a table search field 502, a table list 504, afields selection window 602, and a selected window 604.

In various embodiments, after selection of the ID field, the inputmodule 314 provides a list of available tables of the selected databasein the table list 504. The user may click on a table or search for atable by entering a search query (e.g., a keyword) in the table searchfield 502. Once a table is identified (e.g., clicked on by the user),the fields selection window 506 may provide a list of available fieldsin the selected table. The user may then choose any number of fieldsfrom the fields selection window 602 to be data fields. The selecteddata fields may appear in the selected window 604. The user may alsodeselect fields that appear in the selected window 604.

Those skilled in the art will appreciate that the table selected by theuser in the table list 504 may be the same table selected with regard toFIG. 5. In some embodiments, however, the user may select a differenttable. Further, the user may, in various embodiments, select fields froma variety of different tables.

FIG. 6b is an example metric and filter selection interface window 600 bin some embodiments. The metric and filter selection interface window600 b allows the user to identify a metric, add filter(s), and adjustfilter parameters. The metric and filter selection interface window 600b comprises a metric pull down menu 606, an add filter from databasebutton 608, and an add geometric filter button 610.

In various embodiments, the user may click on the metric pull down menu606 to view a variety of metric options. Various metric options aredescribed herein. In some embodiments, the user may define a metric. Theuser defined metric may then be used with the analysis.

In one example, finite metric space data may be constructed from a datarepository (i.e., database, spreadsheet, or Matlab file). This may meanselecting a collection of fields whose entries will specify the metricusing the standard Euclidean metric for these fields, when they arefloating point or integer variables. Other notions of distance, such asgraph distance between collections of points, may be supported.

The analysis module 320 may perform analysis using the metric as a partof a distance function. The distance function can be expressed by aformula, a distance matrix, or other routine which computes it. The usermay add a filter from a database by clicking on the add filter fromdatabase button 608. The metric space may arise from a relationaldatabase, a Matlab file, an Excel spreadsheet, or other methods forstoring and manipulating data. The metric and filter selection interfacewindow 600 b may allow the user to browse for other filters to use inthe analysis. The analysis and metric function are further describedherein (e.g., see discussion regarding FIG. 8).

The user may also add a geometric filter 610 by clicking on the addgeometric filter button 610. In various embodiments, the metric andfilter selection interface window 600 b may provide a list of geometricfilters from which the user may choose.

FIG. 7 is an example filter parameter interface window 700 in someembodiments. The filter parameter interface window 700 allows the userto determine a resolution for one or more selected filters (e.g.,filters selected in the metric and filter selection interface window600). The filter parameter interface window 700 comprises a filter namemenu 702, an interval field 704, an overlap bar 706, and a done button708.

The filter parameter interface window 700 allows the user to select afilter from the filter name menu 702. In some embodiments, the filtername menu 702 is a drop down box indicating all filters selected by theuser in the metric and filter selection interface window 600. Once afilter is chosen, the name of the filter may appear in the filter namemenu 702. The user may then change the intervals and overlap for one,some, or all selected filters.

The interval field 704 allows the user to define a number of intervalsfor the filter identified in the filter name menu 702. The user mayenter a number of intervals or scroll up or down to get to a desirednumber of intervals. Any number of intervals may be selected by theuser. The function of the intervals is further discussed herein (e.g.,see discussion regarding FIG. 8).

The overlap bar 706 allows the user to define the degree of overlap ofthe intervals for the filter identified in the filter name menu 702. Inone example, the overlap bar 706 includes a slider that allows the userto define the percentage overlap for the interval to be used with theidentified filter. Any percentage overlap may be set by the user.

Once the intervals and overlap are defined for the desired filters, theuser may click the done button. The user may then go back to the metricand filter selection interface window 600 and see a new option to runthe analysis. In some embodiments, the option to run the analysis may beavailable in the filter parameter interface window 700. Once theanalysis is complete, the result may appear in an interactivevisualization further described herein (e.g., see discussion regardingFIGS. 9-11).

It will be appreciated that interface windows in FIGS. 4-7 are examples.The example interface windows are not limited to the functional objects(e.g., buttons, pull down menus, scroll fields, and search fields)shown. Any number of different functional objects may be used. Further,as described herein, any other interface, command line, or graphicaluser interface may be used.

FIG. 8 is a flowchart 800 for data analysis and generating aninteractive visualization in some embodiments. In various embodiments,the processing on data and user-specified options is motivated bytechniques from topology and, in some embodiments, algebraic topology.These techniques may be robust and general. In one example, thesetechniques apply to almost any kind of data for which some qualitativeidea of “closeness” or “similarity” exists. The techniques discussedherein may be robust because the results may be relatively insensitiveto noise in the data and even to errors in the specific details of thequalitative measure of similarity, which, in some embodiments, may begenerally refer to as “the distance function” or “metric.” It will beappreciated that while the description of the algorithms below may seemgeneral, the implementation of techniques described herein may apply toany level of generality.

In step 802, the input module 314 receives data S. In one example, auser identifies a data structure and then identifies ID and data fields.Data S may be based on the information within the ID and data fields. Invarious embodiments, data S is treated as being processed as a finite“similarity space,” where data S has a real-valued function d defined onpairs of points s and tin S, such that:d(s,s)=0d(s,t)=d(t,s)d(s,t)>=0These conditions may be similar to requirements for a finite metricspace, but the conditions may be weaker. In various examples, thefunction is a metric.

It will be appreciated that data S may be a finite metric space, or ageneralization thereof, such as a graph or weighted graph. In someembodiments, data S be specified by a formula, an algorithm, or by adistance matrix which specifies explicitly every pairwise distance.

In step 804, the input module 314 generates reference space R. In oneexample, reference space R may be a well-known metric space (e.g., suchas the real line). The reference space R may be defined by the user. Instep 806, the analysis module 320 generates a map ref( ) from S into R.The map ref( ) from S into R may be called the “reference map.”

In one example, a reference of map from S is to a reference metric spaceR. R may be Euclidean space of some dimension, but it may also be thecircle, torus, a tree, or other metric space. The map can be describedby one or more filters (i.e., real valued functions on S). These filterscan be defined by geometric invariants, such as the output of a densityestimator, a notion of data depth, or functions specified by the originof S as arising from a data set.

In step 808, the resolution module 318 generates a cover of R based onthe resolution received from the user (e.g., filter(s), intervals, andoverlap—see discussion regarding FIG. 7 for example). The cover of R maybe a finite collection of open sets (in the metric of R) such that everypoint in R lies in at least one of these sets. In various examples, R isk-dimensional Euclidean space, where k is the number of filterfunctions. More precisely in this example, R is a box in k-dimensionalEuclidean space given by the product of the intervals [min_k, max_k],where min_k is the minimum value of the k-th filter function on S, andmax_k is the maximum value.

For example, suppose there are 2 filter functions, F1 and F2, and thatF1's values range from −1 to +1, and F2's values range from 0 to 5. Thenthe reference space is the rectangle in the x/y plane with corners(−1,0), (1,0), (−1, 5), (1, 5), as every point s of S will give rise toa pair (F1(s), F2(s)) that lies within that rectangle.

In various embodiments, the cover of R is given by taking products ofintervals of the covers of [min_k,max_k] for each of the k filters. Inone example, if the user requests 2 intervals and a 50% overlap for F1,the cover of the interval [−1,+1] will be the two intervals (−1.5, 0.5),(−0.5, 1.5). If the user requests 5 intervals and a 30% overlap for F2,then that cover of [0, 5] will be (−0.3, 1.3), (0.7, 2.3), (1.7, 3.3),(2.7, 4.3), (3.7, 5.3). These intervals may give rise to a cover of the2-dimensional box by taking all possible pairs of intervals where thefirst of the pair is chosen from the cover for F1 and the second fromthe cover for F2. This may give rise to 2*5, or 10, open boxes thatcovered the 2-dimensional reference space. However, those skilled in theart will appreciate that the intervals may not be uniform, or that thecovers of a k-dimensional box may not be constructed by products ofintervals. In some embodiments, there are many other choices ofintervals. Further, in various embodiments, a wide range of coversand/or more general reference spaces may be used.

In one example, given a cover, C₁, . . . , C_(m), of R, the referencemap is used to assign a set of indices to each point in S, which are theindices of the C_(j) such that ref(s) belongs to C_(j). This functionmay be called ref_tags(s). In a language such as Java, ref_tags would bea method that returned an int[ ]. Since the C's cover R in this example,ref(s) must lie in at least one of them, but the elements of the coverusually overlap one another, which means that points that “land near theedges” may well reside in multiple cover sets. In considering the twofilter example, if F1(s) is −0.99, and F2(s) is 0.001, then ref(s) is(−0.99, 0.001), and this lies in the cover element (−1.5,0.5)×(−0.3,1.3). Supposing that was labeled C₁, the reference map mayassign s to the set {1}. On the other hand, if t is mapped by F1, F2 to(0.1, 2.1), then ref(t) will be in (−1.5, 0.5)×(0.7, 2.3), (−0.5,1.5)×(0.7,2.3), (−1.5,0.5)×(1.7,3.3), and (−0.5, 1.5)×(1.7,3.3), so theset of indices would have four elements for t.

Having computed, for each point, which “cover tags” it is assigned to,for each cover element, C_(d), the points may be constructed, whose tagsincluded, as set S(d). This may mean that every point s is in S(d) forsome d, but some points may belong to more than one such set. In someembodiments, there is, however, no requirement that each S(d) isnon-empty, and it is frequently the case that some of these sets areempty. In the non-parallelized version of some embodiments, each point xis processed in turn, and x is inserted into a hash-bucket for each j inref_tags(t) (that is, this may be how S(d) sets are computed).

It will be appreciated that the cover of the reference space R may becontrolled by the number of intervals and the overlap identified in theresolution (e.g., see further discussion regarding FIG. 7). For example,the more intervals, the finer the resolution in S—that is, the fewerpoints in each S(d), but the more similar (with respect to the filters)these points may be. The greater the overlap, the more times thatclusters in S(d) may intersect clusters in S(e)—this means that more“relationships” between points may appear, but, in some embodiments, thegreater the overlap, the more likely that accidental relationships mayappear.

In step 810, the analysis module 320 clusters each S(d) based on themetric, filter, and the space S. In some embodiments, a dynamicsingle-linkage clustering algorithm may be used to partition S(d). Itwill be appreciated that any number of clustering algorithms may be usedwith embodiments discussed herein. For example, the clustering schememay be k-means clustering for some k, single linkage clustering, averagelinkage clustering, or any method specified by the user.

The significance of the user-specified inputs may now be seen. In someembodiments, a filter may amount to a “forced stretching” in a certaindirection. In some embodiments, the analysis module 320 may not clustertwo points unless ALL of the filter values are sufficiently “related”(recall that while normally related may mean “close,” the cover mayimpose a much more general relationship on the filter values, such asrelating two points s and t if ref(s) and ref(t) are sufficiently closeto the same circle in the plane). In various embodiments, the ability ofa user to impose one or more “critical measures” makes this techniquemore powerful than regular clustering, and the fact that these filterscan be anything, is what makes it so general.

The output may be a simplicial complex, from which one can extract its1-skeleton. The nodes of the complex may be partial clusters, (i.e.,clusters constructed from subsets of S specified as the preimages ofsets in the given covering of the reference space R).

In step 812, the visualization engine 322 identifies nodes which areassociated with a subset of the partition elements of all of the S(d)for generating an interactive visualization. For example, suppose thatS={1, 2, 3, 4}, and the cover is C₁, C₂, C₃. Then if ref_tags(1)={1, 2,3} and ref_tags(2)={2, 3}, and ref_tags(3)={3}, and finallyref_tags(4)={1, 3}, then S(1) in this example is {1, 4}, S(2)={1,2}, andS(3)={1, 2, 3, 4}. If 1 and 2 are close enough to be clustered, and 3and 4 are, but nothing else, then the clustering for S(1) may be {1}{3}, and for S(2) it may be {1, 2}, and for S(3) it may be {1, 2}, {3,4}. So the generated graph has, in this example, at most four nodes,given by the sets {1}, {4}, {1, 2}, and {3, 4} (note that {1, 2} appearsin two different clusterings). Of the sets of points that are used, twonodes intersect provided that the associated node sets have a non-emptyintersection (although this could easily be modified to allow users torequire that the intersection is “large enough” either in absolute orrelative terms).

Nodes may be eliminated for any number of reasons. For example, a nodemay be eliminated as having too few points and/or not being connected toanything else. In some embodiments, the criteria for the elimination ofnodes (if any) may be under user control or have application-specificrequirements imposed on it. For example, if the points are consumers,for instance, clusters with too few people in area codes served by acompany could be eliminated. If a cluster was found with “enough”customers, however, this might indicate that expansion into area codesof the other consumers in the cluster could be warranted.

In step 814, the visualization engine 322 joins clusters to identifyedges (e.g., connecting lines between nodes). Once the nodes areconstructed, the intersections (e.g., edges) may be computed “all atonce,” by computing, for each point, the set of node sets (not ref_tags,this time). That is, for each s in S, node_id_set(s) may be computed,which is an int[ ]. In some embodiments, if the cover is well behaved,then this operation is linear in the size of the set S, and we theniterate over each pair in node_id_set(s). There may be an edge betweentwo node_id's if they both belong to the same node_id_set( ) value, andthe number of points in the intersection is precisely the number ofdifferent node_id sets in which that pair is seen. This means that,except for the clustering step (which is often quadratic in the size ofthe sets S(d), but whose size may be controlled by the choice of cover),all of the other steps in the graph construction algorithm may be linearin the size of S, and may be computed quite efficiently.

In step 816, the visualization engine 322 generates the interactivevisualization of interconnected nodes (e.g., nodes and edges displayedin FIGS. 9 and 10).

It will be appreciated that it is possible, in some embodiments, to makesense in a fairly deep way of connections between various ref( ) mapsand/or choices of clustering. Further, in addition to computing edges(pairs of nodes), the embodiments described herein may be extended tocompute triples of nodes, etc. For example, the analysis module 320 maycompute simplicial complexes of any dimension (by a variety of rules) onnodes, and apply techniques from homology theory to the graphs to helpusers understand a structure in an automatic (or semi-automatic) way.

Further, it will be appreciated that uniform intervals in the coveringmay not always be a good choice. For example, if the points areexponentially distributed with respect to a given filter, uniformintervals can fail—in such case adaptive interval sizing may yielduniformly-sized S(d) sets, for instance.

Further, in various embodiments, an interface may be used to encodetechniques for incorporating third-party extensions to data access anddisplay techniques. Further, an interface may be used to for third-partyextensions to underlying infrastructure to allow for new methods forgenerating coverings, and defining new reference spaces.

FIG. 9 is an example interactive visualization 900 in some embodiments.The display of the interactive visualization may be considered a “graph”in the mathematical sense. The interactive visualization comprises oftwo types of objects: nodes (e.g., nodes 902 and 906) (which may beballs and may be colored) and the edges (e.g., edge 904) (the blacklines). The edges connect pairs of nodes (e.g., edge 904 connects node902 with node 906). As discussed herein, each node may represent acollection of data points (rows in the database identified by the user).In one example, connected nodes tend to include data points which are“similar to” (e.g., clustered with) each other. The collection of datapoints may be referred to as being “in the node.” The interactivevisualization may be two-dimensional, three-dimensional, or acombination of both.

In various embodiments, connected nodes and edges may form a graph orstructure. There may be multiple graphs in the interactivevisualization. In one example, the interactive visualization may displaytwo or more unconnected structures of nodes and edges.

The visual properties of the nodes and edges (such as, but not limitedto, color, stroke color, text, texture, shape, coordinates of the nodeson the screen) can encode any data based property of the data pointswithin each node. For example, coloring of the nodes and/or the edgesmay indicate (but is not limited to) the following:

-   -   Values of fields or filters    -   Any general functions of the data in the nodes (e.g., if the        data were unemployment rates by state, then GDP of the states        may be identifiable by color the nodes)    -   Number of data points in the node

The interactive visualization 900 may contain a “bar” 910 which maycomprise a legend indicating patterns and/or coloring of the nodes(e.g., balls) and may also identify what the patterns and/or colorsindicate. For example, in FIG. 9, bar 910 may indicate that color ofsome nodes is based on the density filter with blue (on the far left ofthe bar 910) indicating “4.99e+03” and red (on the far right of the bar910) indicating “1.43e+04.” In general this might be expanded to showany other legend by which nodes and/or edges are colored. It will beappreciated that, in some embodiments, the user may control the color aswell as what the color (and/or stroke color, text, texture, shape,coordinates of the nodes on the screen) indicates.

The user may also drag and drop objects of the interactive visualization900. In various embodiments, the user may reorient structures of nodesand edges by dragging one or more nodes to another portion of theinteractive visualization (e.g., a window). In one example, the user mayselect node 902, hold node 902, and drag the node across the window. Thenode 902 will follow the user's cursor, dragging the structure of edgesand/or nodes either directly or indirectly connected to the node 902. Insome embodiments, the interactive visualization 900 may depict multipleunconnected structures. Each structure may include nodes, however, noneof the nodes of either structure are connected to each other. If theuser selects and drags a node of the first structure, only the firststructure will be reoriented with respect to the user action. The otherstructure will remain unchanged. The user may wish to reorient thestructure in order to view nodes, select nodes, and/or better understandthe relationships of the underlying data.

In one example, a user may drag a node to reorient the interactivevisualization (e.g., reorient the structure of nodes and edges). Whilethe user selects and/or drags the node, the nodes of the structureassociated with the selected node may move apart from each other inorder to provide greater visibility. Once the user lets go (e.g.,deselects or drops the node that was dragged), the nodes of thestructure may continue to move apart from each other.

In various embodiments, once the visualization engine 322 generates theinteractive display, the depicted structures may move by spreading outthe nodes from each other. In one example, the nodes spread from eachother slowly allowing the user to view nodes distinguish from each otheras well as the edges. In some embodiments, the visualization engine 322optimizes the spread of the nodes for the user's view. In one example,the structure(s) stop moving once an optimal view has been reached.

It will be appreciated that the interactive visualization 900 mayrespond to gestures (e.g., multi-touch), stylus, or other interactionsallowing the user to reorient nodes and edges and/or interacting withthe underlying data.

The interactive visualization 900 may also respond to user actions suchas when the user drags, clicks, or hovers a mouse cursor over a node. Insome embodiments, when the user selects a node or edge, node informationor edge information may be displayed. In one example, when a node isselected (e.g., clicked on by a user with a mouse or a mouse cursorhovers over the node), a node information box 908 may appear thatindicates information regarding the selected node. In this example, thenode information box 908 indicates an ID, box ID, number of elements(e.g., data points associated with the node), and density of the dataassociated with the node.

The user may also select multiple nodes and/or edges by clickingseparate on each object, or drawing a shape (such as a box) around thedesired objects. Once the objects are selected, a selection informationbox 912 may display some information regarding the selection. Forexample, selection information box 912 indicates the number of nodesselected and the total points (e.g., data points or elements) of theselected nodes.

The interactive visualization 900 may also allow a user to furtherinteract with the display. Color option 914 allows the user to displaydifferent information based on color of the objects. Color option 914 inFIG. 9 is set to filter_Density, however, other filters may be chosenand the objects re-colored based on the selection. It will beappreciated that the objects may be colored based on any filter,property of data, or characterization. When a new option is chosen inthe color option 914, the information and/or colors depicted in thecolor bar 910 may be updated to reflect the change.

Layout checkbox 916 may allow the user to anchor the interactivevisualization 900. In one example, the layout checkbox 916 is checkedindicating that the interactive visualization 900 is anchored. As aresult, the user will not be able to select and drag the node and/orrelated structure. Although other functions may still be available, thelayout checkbox 916 may help the user keep from accidentally movingand/or reorienting nodes, edges, and/or related structures. It will beappreciated the layout checkbox 916 may indicate that the interactivevisualization 900 is anchored when the layout checkbox 916 is uncheckedand that when the layout checkbox 916 is checked the interactivevisualization 900 is no longer anchored.

The change parameters button 918 may allow a user to change theparameters (e.g., add/remove filters and/or change the resolution of oneor more filters). In one example, when the change parameters button 918is activated, the user may be directed back to the metric and filterselection interface window 600 (see FIG. 6) which allows the user to addor remove filters (or change the metric). The user may then view thefilter parameter interface 700 (see FIG. 7) and change parameters (e.g.,intervals and overlap) for one or more filters. The analysis module 320may then re-analyze the data based on the changes and display a newinteractive visualization 900 without again having to specify the datasets, filters, etc.

The find ID's button 920 may allow a user to search for data within theinteractive visualization 900. In one example, the user may click thefind ID's button 920 and receive a window allowing the user to identifydata or identify a range of data. Data may be identified by ID orsearching for the data based on properties of data and/or metadata. Ifdata is found and selected, the interactive visualization 900 mayhighlight the nodes associated with the selected data. For example,selecting a single row or collection of rows of a database orspreadsheet may produce a highlighting of nodes whose correspondingpartial cluster contains any element of that selection.

In various embodiments, the user may select one or more objects andclick on the explain button 922 to receive in-depth informationregarding the selection. In some embodiments, when the user selects theexplain button 922, the information about the data from which theselection is based may be displayed. The function of the explain button922 is further discussed herein (e.g., see discussion regarding FIG.10).

In various embodiments, the interactive visualization 900 may allow theuser to specify and identify subsets of interest, such as outputfiltering, to remove clusters or connections which are too small orotherwise uninteresting. Further, the interactive visualization 900 mayprovide more general coloring and display techniques, including, forexample, allowing a user to highlight nodes based on a user-specifiedpredicate, and coloring the nodes based on the intensity ofuser-specified weighting functions.

The interactive visualization 900 may comprise any number of menu items.The “Selection” menu may allow the following functions:

-   -   Select singletons (select nodes which are not connected to other        nodes)    -   Select all (selects all the nodes and edges)    -   Select all nodes (selects all nodes)    -   Select all edges    -   Clear selection (no selection)    -   Invert Selection (selects the complementary set of nodes or        edges)    -   Select “small” nodes (allows the user to threshold nodes based        on how many points they have)    -   Select leaves (selects all nodes which are connected to long        “chains” in the graph)    -   Remove selected nodes    -   Show in a table (shows the selected nodes and their associated        data in a table)    -   Save selected nodes (saves the selected data to whatever format        the user chooses. This may allow the user to subset the data and        create new data sources which may be used for further analysis.)

In one example of the “show in a table” option, information from aselection of nodes may be displayed. The information may be specific tothe origin of the data. In various embodiments, elements of a databasetable may be listed, however, other methods specified by the user mayalso be included. For example, in the case of microarray data from geneexpression data, heat maps may be used to view the results of theselections.

The interactive visualization 900 may comprise any number of menu items.The “Save” menu may allow may allow the user to save the whole output ina variety of different formats such as (but not limited to):

-   -   Image files (PNG/JPG/PDF/SVG etc.)    -   Binary output (The interactive output is saved in the binary        format. The user may reopen this file at any time to get this        interactive window again)        In some embodiments, graphs may be saved in a format such that        the graphs may be used for presentations. This may include        simply saving the image as a pdf or png file, but it may also        mean saving an executable .xml file, which may permit other        users to use the search and save capability to the database on        the file without having to recreate the analysis.

In various embodiments, a relationship between a first and a secondanalysis output/interactive visualization for differing values of theinterval length and overlap percentage may be displayed. The formalrelationship between the first and second analysis output/interactivevisualization may be that when one cover refines the next, there is amap of simplicial complexes from the output of the first to the outputof the second. This can be displayed by applying a restricted form of athree-dimensional graph embedding algorithm, in which a graph is theunion of the graphs for the various parameter values and in which theconnections are the connections in the individual graphs as well asconnections from one node to its image in the following graph. Theconstituent graphs may be placed in its own plane in 3D space. In someembodiments, there is a restriction that each constituent graph remainwithin its associated plane. Each constituent graph may be displayedindividually, but a small change of parameter value may result in thevisualization of the adjacent constituent graph. In some embodiments,nodes in the initial graph will move to nodes in the next graph, in areadily visualizable way.

FIG. 10 is an example interactive visualization 1000 displaying anexplain information window 1002 in some embodiments. In variousembodiments, the user may select a plurality of nodes and click on theexplain button. When the explain button is clicked, the explaininformation window 1002 may be generated. The explain information window1002 may identify the data associated with the selected object(s) aswell as information (e.g., statistical information) associated with thedata.

In some embodiments, the explain button allows the user to get a sensefor which fields within the selected data fields are responsible for“similarity” of data in the selected nodes and the differentiatingcharacteristics. There can be many ways of scoring the data fields. Theexplain information window 1002 (i.e., the scoring window in FIG. 10) isshown along with the selected nodes. The highest scoring fields maydistinguish variables with respect to the rest of the data.

In one example, the explain information window 1002 indicates that datafrom fields day0-day6 has been selected. The minimum value of the datain all of the fields is 0. The explain information window 1002 alsoindicates the maximum values. For example, the maximum value of all ofthe data associated with the day0 field across all of the points of theselected nodes is 0.353. The average (i.e., mean) of all of the dataassociated with the day0 field across all of the points of the selectednodes is 0.031. The score may be a relative (e.g., normalized) valueindicating the relative function of the filter; here, the score mayindicate the relative density of the data associated with the day0 fieldacross all of the points of the selected nodes. Those skilled in the artwill appreciate that any information regarding the data and/or selectednodes may appear in the explain information window 1002.

It will be appreciated that the data and the interactive visualization1000 may be interacted with in any number of ways. The user may interactwith the data directly to see where the graph corresponds to the data,make changes to the analysis and view the changes in the graph, modifythe graph and view changes to the data, or perform any kind ofinteraction.

FIG. 11 is a flowchart 1100 of functionality of the interactivevisualization in some embodiments. In step 1102, the visualizationengine 322 receives the analysis from the analysis module 320 and graphsnodes as balls and edges as connectors between balls 1202 to createinteractive visualization 900 (see FIG. 9).

In step 1104, the visualization engine 322 determines if the user ishovering a mouse cursor over (or has selected) a ball (i.e., a node). Ifthe user is hovering a mouse cursor over a ball or is selecting a ball,then information may be displayed regarding the data associated with theball. In one example, the visualization engine 322 displays a nodeinformation window 908.

If the visualization engine 322 does not determine that the user ishovering a mouse cursor over (or has selected) a ball, then thevisualization engine 322 determines if the user has selected balls onthe graph (e.g., by clicking on a plurality of balls or drawing a boxaround a plurality of balls). If the user has selected a plurality ofballs on the graph, the visualization engine 322 may highlight theselected balls on the graph in step 1110. The visualization engine 322may also display information regarding the selection (e.g., bydisplaying a selection information window 912). The user may also clickon the explain button 922 to receive more information associated withthe selection (e.g., the visualization engine 322 may display theexplain information window 1002).

In step 1112, the user may save the selection. For example, thevisualization engine 322 may save the underlying data, selected metric,filters, and/or resolution. The user may then access the savedinformation and create a new structure in another interactivevisualization 900 thereby allowing the user to focus attention on asubset of the data.

If the visualization engine 322 does not determine that the user hasselected balls on the graph, the visualization engine 322 may determineif the user selects and drags a ball on the graph in step 1114. If theuser selects and drags a ball on the graph, the visualization engine 322may reorient the selected balls and any connected edges and balls basedon the user's action in step 1116. The user may reorient all or part ofthe structure at any level of granularity.

It will be appreciated that although FIG. 11 discussed the user hoveringover, selecting, and/or dragging a ball, the user may interact with anyobject in the interactive visualization 900 (e.g., the user may hoverover, select, and/or drag an edge). The user may also zoom in or zoomout using the interactive visualization 900 to focus on all or a part ofthe structure (e.g., one or more balls and/or edges). Any number ofactions and operations may be performed using the interactivevisualization 900.

Further, although balls are discussed and depicted in FIGS. 9-11, itwill be appreciated that the nodes may be any shape and appear as anykind of object. Further, although some embodiments described hereindiscuss an interactive visualization being generated based on the outputof algebraic topology, the interactive visualization may be generatedbased on any kind of analysis and is not limited.

For years, researchers have been collecting huge amounts of data onbreast cancer, yet we are still battling the disease. Complexity, ratherthan quantity, is one of the fundamental issues in extracting knowledgefrom data. A topological data exploration and visualization platform mayassist the analysis and assessment of complex data. In variousembodiments, a predictive and visual cancer map generated by thetopological data exploration and visualization platform may assistphysicians to determine treatment options.

In one example, a breast cancer map visualization may be generated basedon the large amount of available information already generated by manyresearchers. Physicians may send biopsy data directly to a cloud-basedserver which may localize a new patient's data within the breast cancermap visualization. The breast cancer map visualization may be annotated(e.g., labeled) such that the physician may view outcomes of patientswith similar profiles as well as different kinds of statisticalinformation such as survival probabilities. Each new data point from apatient may be incorporated into the breast cancer map visualization toimprove accuracy of the breast cancer map visualization over time.

Although the following examples are largely focused on cancer mapvisualizations, it will be appreciated that at least some of theembodiments described herein may apply to any biological condition andnot be limited to cancer and/or disease. For example, some embodiments,may apply to different industries.

FIG. 12 is a flowchart for generating a cancer map visualizationutilizing biological data of a plurality of patients in someembodiments. In various embodiments, the processing of data anduser-specified options is motivated by techniques from topology and, insome embodiments, algebraic topology. As discussed herein, thesetechniques may be robust and general. In one example, these techniquesapply to almost any kind of data for which some qualitative idea of“closeness” or “similarity” exists. It will be appreciated that theimplementation of techniques described herein may apply to any level ofgenerality.

In various embodiments, a cancer map visualization is generated usinggenomic data linked to clinical outcomes (i.e., medical characteristics)which may be used by physicians during diagnosis and/or treatment.Initially, publicly available data sets may be integrated to constructthe topological map visualizations of patients (e.g., breast cancerpatients). It will be appreciated that any private, public, orcombination of private and public data sets may be integrated toconstruct the topological map visualizations. A map visualization may bebased on biological data such as, but not limited to, gene expression,sequencing, and copy number variation. As such, the map visualizationmay comprise many patients with many different types of collected data.Unlike traditional methods of analysis where distinct studies of breastcancer appear as separate entities, the map visualization may fusedisparate data sets while utilizing many datasets and data types.

In various embodiments, a new patient may be localized on the mapvisualization. With the map visualization for subtypes of a particulardisease and a new patient diagnosed with the disease, point(s) may belocated among the data points used in computing the map visualization(e.g., nearest neighbor) which is closest to the new patient point. Thenew patient may be labeled with nodes in the map visualizationcontaining the closest neighbor. These nodes may be highlighted to givea physician the location of the new patient among the patients in thereference data set. The highlighted nodes may also give the physicianthe location of the new patient relative to annotated disease subtypes.

The visualization map may be interactive and/or searchable in real-timethereby potentially enabling extended analysis and providing speedyinsight into treatment.

In step 1202, biological data and clinical outcomes of previous patientsmay be received. The clinical outcomes may be medical characteristics.Biological data is any data that may represent a condition (e.g., amedical condition) of a person. Biological data may include any healthrelated, medical, physical, physiological, pharmaceutical dataassociated with one or more patients. In one example, biological datamay include measurements of gene expressions for any number of genes. Inanother example, biological data may include sequencing information(e.g., RNA sequencing).

In various embodiments, biological data for a plurality of patients maybe publicly available. For example, various medical health facilitiesand/or public entities may provide gene expression data for a variety ofpatients. In addition to the biological data, information regarding anynumber of clinical outcomes, treatments, therapies, diagnoses and/orprognoses may also be provided. Those skilled in the art will appreciatethat any kind of information may be provided in addition to thebiological data.

The biological data, in one example, may be similar to data S asdiscussed with regard to step 802 of FIG. 8. The biological data mayinclude ID fields that identify patients and data fields that arerelated to the biological information (e.g., gene expressionmeasurements).

FIG. 13 is an example data structure 1300 including biological data 1304a-1304 y for a number of patients 1308 a-1308 n that may be used togenerate the cancer map visualization in some embodiments. Column 1302represents different patient identifiers for different patients. Thepatient identifiers may be any identifier.

At least some biological data may be contained within gene expressionmeasurements 1304 a-1304 y. In FIG. 13, “y” represents any number. Forexample, there may be 50,000 or more separate columns for different geneexpressions related to a single patient or related to one or moresamples from a patient. It will be appreciated that column 1304 a mayrepresent a gene expression measurement for each patient (if any forsome patients) associated with the patient identifiers in column 1302.The column 1304 b may represent a gene expression measurement of one ormore genes that are different than that of column 1304 a. As discussed,there may be any number of columns representing different geneexpression measurements.

Column 1306 may include any number of clinical outcomes, prognoses,diagnoses, reactions, treatments, and/or any other informationassociated with each patient. All or some of the information containedin column 1306 may be displayed (e.g., by a label or an annotation thatis displayed on the visualization or available to the user of thevisualization via clicking) on or for the visualization.

Rows 1308 a-1308 n each contains biological data associated with thepatient identifier of the row. For example, gene expressions in row 1308a are associated with patient identifier P1. As similarly discussed withregard to “y” herein, “n” represents any number. For example, there maybe 100,000 or more separate rows for different patients.

It will be appreciated that there may be any number of data structuresthat contain any amount of biological data for any number of patients.The data structure(s) may be utilized to generate any number of mapvisualizations.

In step 1204, the analysis server may receive a filter selection. Insome embodiments, the filter selection is a density estimation function.It will be appreciated that the filter selection may include a selectionof one or more functions to generate a reference space.

In step 1206, the analysis server performs the selected filter(s) on thebiological data of the previous patients to map the biological data intoa reference space. In one example, a density estimation function, whichis well known in the art, may be performed on the biological data (e.g.,data associated with gene expression measurement data 1304 a-1304 y) torelate each patient identifier to one or more locations in the referencespace (e.g., on a real line).

In step 1208, the analysis server may receive a resolution selection.The resolution may be utilized to identify overlapping portions of thereference space (e.g., a cover of the reference space R) in step 1210.

As discussed herein, the cover of R may be a finite collection of opensets (in the metric of R) such that every point in R lies in at leastone of these sets. In various examples, R is k-dimensional Euclideanspace, where k is the number of filter functions. Those skilled in theart will appreciate that the cover of the reference space R may becontrolled by the number of intervals and the overlap identified in theresolution (e.g., see FIG. 7). For example, the more intervals, thefiner the resolution in S (e.g., the similarity space of the receivedbiological data)—that is, the fewer points in each S(d), but the moresimilar (with respect to the filters) these points may be. The greaterthe overlap, the more times that clusters in S(d) may intersect clustersin S(e)—this means that more “relationships” between points may appear,but, in some embodiments, the greater the overlap, the more likely thataccidental relationships may appear.

In step 1212, the analysis server receives a metric to cluster theinformation of the cover in the reference space to partition S(d). Inone example, the metric may be a Pearson Correlation. The clusters mayform the groupings (e.g., nodes or balls). Various cluster means may beused including, but not limited to, a single linkage, average linkage,complete linkage, or k-means method.

As discussed herein, in some embodiments, the analysis module 320 maynot cluster two points unless filter values are sufficiently “related”(recall that while normally related may mean “close,” the cover mayimpose a much more general relationship on the filter values, such asrelating two points s and t if ref(s) and ref(t) are sufficiently closeto the same circle in the plane where ref( ) represents one or morefilter functions). The output may be a simplicial complex, from whichone can extract its 1-skeleton. The nodes of the complex may be partialclusters, (i.e., clusters constructed from subsets of S specified as thepreimages of sets in the given covering of the reference space R).

In step 1214, the analysis server may generate the visualization mapwith nodes representing clusters of patient members and edges betweennodes representing common patient members. In one example, the analysisserver identifies nodes which are associated with a subset of thepartition elements of all of the S(d) for generating an interactivevisualization.

As discussed herein, for example, suppose that S={1, 2, 3, 4}, and thecover is C₁, C₂, C₃. Suppose cover C₁ contains {1, 4}, C₂ contains {1,2}, and C₃ contains {1, 2, 3, 4}. If 1 and 2 are close enough to beclustered, and 3 and 4 are, but nothing else, then the clustering forS(1) may be {1}, {4}, and for S(2) it may be {1, 2}, and for S(3) it maybe {1, 2}, {3, 4}. So the generated graph has, in this example, at mostfour nodes, given by the sets {1}, {4}, {1, 2}, and {3, 4} (note that{1, 2} appears in two different clusterings). Of the sets of points thatare used, two nodes intersect provided that the associated node setshave a non-empty intersection (although this could easily be modified toallow users to require that the intersection is “large enough” either inabsolute or relative terms).

As a result of clustering, member patients of a grouping may sharebiological similarities (e.g., similarities based on the biologicaldata).

The analysis server may join clusters to identify edges (e.g.,connecting lines between nodes). Clusters joined by edges (i.e.,interconnections) share one or more member patients. In step 1216, adisplay may display a visualization map with attributes based on theclinical outcomes contained in the data structures (e.g., see FIG. 13regarding clinical outcomes). Any labels or annotations may be utilizedbased on information contained in the data structures. For example,treatments, prognoses, therapies, diagnoses, and the like may be used tolabel the visualization. In some embodiments, the physician or otheruser of the map visualization accesses the annotations or labels byinteracting with the map visualization.

The resulting cancer map visualization may reveal interactions andrelationships that were obscured, untested, and/or previously notrecognized.

FIG. 14 is an example visualization displaying the cancer mapvisualization 1400 in some embodiments. The cancer map visualization1400 represents a topological network of cancer patients. The cancer mapvisualization 1400 may be based on publicly and/or privately availabledata.

In various embodiments, the cancer map visualization 1400 is createdusing gene expression profiles of excised tumors. Each node (i.e., ballor grouping displayed in the map visualization 1400) contains a subsetof patients with similar genetic profiles.

As discussed herein, one or more patients (i.e., patient members of eachnode or grouping) may occur in multiple nodes. A patient may share asimilar genetic profile with multiple nodes or multiple groupings. Inone example, of 50,000 different gene expressions of the biologicaldata, multiple patients may share a different genetic profiles (e.g.,based on different gene expression combinations) with differentgroupings. When a patient shares a similar genetic profile withdifferent groupings or nodes, the patient may be included within thegroupings or nodes.

The cancer map visualization 1400 comprises groupings andinterconnections that are associated with different clinical outcomes.All or some of the clinical outcomes may be associated with thebiological data that generated the cancer map visualization 1400. Thecancer map visualization 1400 includes groupings associated withsurvivors 1402 and groupings associated with non-survivors 1404. Thecancer map visualization 1400 also includes different groupingsassociated with estrogen receptor positive non-survivors 1406, estrogenreceptor negative non-survivors 1408, estrogen receptor positivesurvivors 1410, and estrogen receptor negative survivors 1412.

In various embodiments, when one or more patients are members of two ormore different nodes, the nodes are interconnected by an edge (e.g., aline or interconnection). If there is not an edge between the two nodes,then there are no common member patients between the two nodes. Forexample, grouping 1414 shares at least one common member patient withgrouping 1418. The intersection of the two groupings is represented byedge 1416. As discussed herein, the number of shared member patients ofthe two groupings may be represented in any number of ways includingcolor of the interconnection, color of the groupings, size of theinterconnection, size of the groupings, animations of theinterconnection, animations of the groupings, brightness, or the like.In some embodiments, the number and/or identifiers of shared memberpatients of the two groupings may be available if the user interactswith the groupings 1414 and/or 1418 (e.g., draws a box around the twogroupings and the interconnection utilizing an input device such as amouse).

In various embodiments, a physician, on obtaining some data on a breasttumor, direct the data to an analysis server (e.g., analysis server 208over a network such as the Internet) which may localize the patientrelative to one or more groupings on the cancer map visualization 1400.The context of the cancer map visualization 1400 may enable thephysician to assess various possible outcomes (e.g., proximity ofrepresentation of new patient to the different associations of clinicaloutcomes).

FIG. 15 is a flowchart of for positioning new patient data relative to acancer map visualization in some embodiments. In step 1502, newbiological data of a new patient is received. In various embodiments, aninput module 314 of an analysis server (e.g., analysis server 208 ofFIGS. 1 and 2) may receive biological data of a new patient from aphysician or medical facility that performed analysis of one or moresamples to generate the biological data. The biological data may be anydata that represents a biological data of the new patient including, forexample, gene expressions, sequencing information, or the like.

In some embodiments, the analysis server 208 may comprise a new patientdistance module and a location engine. In step 1504, the new patientdistance module determines distances between the biological data of eachpatient of the cancer map visualization 1600 and the new biological datafrom the new patient. For example, the previous biological data that wasutilized in the generation of the cancer map visualization 1600 may bestored in mapped data structures. Distances may be determined betweenthe new biological data of the new patient and each of the previouspatient's biological data in the mapped data structure.

It will be appreciated that distances may be determined in any number ofways using any number of different metrics or functions. Distances maybe determined between the biological data of the previous patients andthe new patients. For example, a distance may be determined between afirst gene expression measurement of the new patient and each (or asubset) of the first gene expression measurements of the previouspatients (e.g., the distance between G1 of the new patient and G1 ofeach previous patient may be calculated). Distances may be determinedbetween all (or a subset of) other gene expression measurements of thenew patient to the gene expression measurements of the previouspatients.

In various embodiments, a location of the new patient on the cancer mapvisualization 1600 may be determined relative to the other memberpatients utilizing the determined distances.

In step 1506, the new patient distance module may compare distancesbetween the patient members of each grouping to the distances determinedfor the new patient. The new patient may be located in the grouping ofpatient members that are closest in distance to the new patient. In someembodiments, the new patient location may be determined to be within agrouping that contains the one or more patient members that are closestto the new patient (even if other members of the grouping have longerdistances with the new patient). In some embodiments, this step isoptional.

In various embodiments, a representative patient member may bedetermined for each grouping. For example, some or all of the patientmembers of a grouping may be averaged or otherwise combined to generatea representative patient member of the grouping (e.g., the distancesand/or biological data of the patient members may be averaged oraggregated). Distances may be determined between the new patientbiological data and the averaged or combined biological data of one ormore representative patient members of one or more groupings. Thelocation engine may determine the location of the new patient based onthe distances. In some embodiments, once the closest distance betweenthe new patient and the representative patient member is found,distances may be determined between the new patient and the individualpatient members of the grouping associated with the closestrepresentative patient member.

In optional step 1508, a diameter of the grouping with the one or moreof the patient members that are closest to the new patient (based on thedetermined distances) may be determined. In one example, the diametersof the groupings of patient members closest to the new patient arecalculated. The diameter of the grouping may be a distance between twopatient members who are the farthest from each other when compared tothe distances between all patient members of the grouping. If thedistance between the new patient and the closest patient member of thegrouping is less than the diameter of the grouping, the new patient maybe located within the grouping. If the distance between the new patientand the closest patient member of the grouping is greater than thediameter of the grouping, the new patient may be outside the grouping(e.g., a new grouping may be displayed on the cancer map visualizationwith the new patient as the single patient member of the grouping). Ifthe distance between the new patient and the closest patient member ofthe grouping is equal to the diameter of the grouping, the new patientmay be placed within or outside the grouping.

It will be appreciated that the determination of the diameter of thegrouping is not required in determining whether the new patient locationis within or outside of a grouping. In various embodiments, adistribution of distances between member patients and between memberpatients and the new patient is determined. The decision to locate thenew patient within or outside of the grouping may be based on thedistribution. For example, if there is a gap in the distribution ofdistances, the new patient may be separated from the grouping (e.g., asa new grouping). In some embodiments, if the gap is greater than apreexisting threshold (e.g., established by the physician, other user,or previously programmed), the new patient may be placed in a newgrouping that is placed relative to the grouping of the closest memberpatients. The process of calculating the distribution of distances ofcandidate member patients to determine whether there may be two or moregroupings may be utilized in generation of the cancer map visualizationfurther described herein (e.g., in the process as described with regardto FIG. 12). It will be appreciated that there may be any number of waysto determine whether a new patient should be included within a groupingof other patient members.

In step 1510, the location engine determines the location of the newpatient relative to the member patients and/or groupings of the cancermap visualization. The new location may be relative to the determineddistances between the new patient and the previous patients. Thelocation of the new patient may be part of a previously existinggrouping or may form a new grouping.

In some embodiments, the location of the new patient with regard to thecancer map visualization may be performed locally to the physician. Forexample, the cancer map visualization 1400 may be provided to thephysician (e.g., via a digital device). The physician may load the newpatient's biological data locally and the distances may be determinedlocally or via a cloud-based server. The location(s) associated with thenew patient may be overlaid on the previously existing cancer mapvisualization either locally or remotely.

It will be appreciated that, in some embodiments, the previous state ofthe cancer map visualization (e.g., cancer map visualization 1400) maybe retained or otherwise stored and a new cancer map visualizationgenerated utilizing the new patient biological data (e.g., in a methodsimilar to that discussed with regard to FIG. 12). The newly generatedmap may be compared to the previous state and the differences may behighlighted thereby, in some embodiments, highlighting the location(s)associated with the new patient. In this way, distances may be not becalculated as described with regard to FIG. 15, but rather, the processmay be similar to that as previously discussed.

FIG. 16 is an example visualization displaying the cancer map includingpositions for three new cancer patients in some embodiments. The cancermap visualization 1400 comprises groupings and interconnections that areassociated with different clinical outcomes as discussed with regard toFIG. 14. All or some of the clinical outcomes may be associated with thebiological data that generated the cancer map visualization 1400. Thecancer map visualization 1400 includes different groupings associatedwith survivors 1402, groupings associated with non-survivors 1404,estrogen receptor positive non-survivors 1406, estrogen receptornegative non-survivors 1408, estrogen receptor positive survivors 1410,and estrogen receptor negative survivors 1412.

The cancer map visualization 1400 includes three locations for three newbreast cancer patients. The breast cancer patient location 1602 isassociated with the clinical outcome of estrogen receptor positivesurvivors. The breast cancer patient location 1604 is associated withthe clinical outcome of estrogen receptor negative survivors.Unfortunately, breast cancer patient location 1606 is associated withestrogen receptor negative non-survivors. Based on the locations, aphysician may consider different diagnoses, prognoses, treatments, andtherapies to maintain or attempt to move the breast cancer patient to adifferent location utilizing the cancer map visualization 1400.

In some embodiments, the physician may assess the underlying biologicaldata associated with any number of member patients of any number ofgroupings to better understand the genetic similarities and/ordissimilarities. The physician may utilize the information to makebetter informed decisions.

The patient location 1604 is highlighted on the cancer map visualization1400 as active (e.g., selected by the physician). It will be appreciatedthat the different locations may be of any color, size, brightness,and/or animated to highlight the desired location(s) for the physician.Further, although only one location is identified for three differentbreast cancer patients, any of the breast cancer patients may havemultiple locations indicating different genetic similarities.

It will be appreciated that the cancer map visualization 1400 may beupdated with new information at any time. As such, as new patients areadded to the cancer map visualization 1400, the new data updates thevisualization such that as future patients are placed in the map, themap may already include the updated information. As new informationand/or new patient data is added to the cancer map visualization 1400,the cancer map visualization 1400 may improve as a tool to better informphysicians or other medical professionals.

In various embodiments, the cancer map visualization 1400 may trackchanges in patients over time. For example, updates to a new patient maybe visually tracked as changes in are measured in the new patient'sbiological data. In some embodiments, previous patient data is similarlytracked which may be used to determine similarities of changes based oncondition, treatment, and/or therapies, for example. In variousembodiments, velocity of change and/or acceleration of change of anynumber of patients may be tracked over time using or as depicted on thecancer map visualization 1400. Such depictions may assist the treatingphysician or other personnel related to the treating physician to betterunderstand changes in the patient and provide improved, current, and/orupdated diagnoses, prognoses, treatments, and/or therapies.

FIG. 17 is a flowchart of utilization the visualization and positioningof new patient data in some embodiments. In various embodiments, aphysician may collect amounts of genomic information from tumors removedfrom a new patient, input the data (e.g., upload the data to an analysisserver), and receive a map visualization with a location of the newpatient. The new patient's location within the map may offer thephysician new information about the similarities to other patients. Insome embodiments, the map visualization may be annotated so that thephysician may check the outcomes of previous patients in a given regionof the map visualization are distributed and then use the information toassist in decision-making for diagnosis, treatment, prognosis, and/ortherapy.

In step 1702, a medical professional or other personnel may remove asample from a patient. The sample may be of a tumor, blood, or any otherbiological material. In one example, a medical professional performs atumor excision. Any number of samples may be taken from a patient.

In step 1704, the sample(s) may be provided to a medical facility todetermine new patient biological data. In one example, the medicalfacility measures genomic data such as gene expression of a number ofgenes or protein levels.

In step 1706, the medical professional or other entity associated withthe medical professional may receive the new patient biological databased on the sample(s) from the new patient. In one example, a physicianmay receive the new patient biological data. The physician may provideall or some of the new patient biological data to an analysis serverover the Internet (e.g., the analysis server may be a cloud-basedserver). In some embodiments, the analysis server is the analysis server208 of FIG. 2. In some embodiments, the medical facility that determinesthe new patient biological data provides the biological data in anelectronic format which may be uploaded to the analysis server. In someembodiments, the medical facility that determines the new patientbiological data (e.g., the medical facility that measures the genomicdata) provide the biological data to the analysis server at the requestof the physician or others associated with the physician. It will beappreciated that the biological data may be provided to the analysisserver in any number of ways.

The analysis server may be any digital device and may not be limited toa digital device on a network. In some embodiments, the physician mayhave access to the digital device. For example, the analysis server maybe a table, personal computer, local server, or any other digitaldevice.

Once the analysis server receives the biological data of the new patient(e.g., the new patient biological data may be uploaded to the analysisserer in step 1708), the new patient may be localized in the mapvisualization and the information may be sent back to the physician instep 1710. The visualization may be a map with nodes representingclusters of previous patient members and edges between nodesrepresenting common patient members. The visualization may furtherdepict one or more locations related to the biological data of the newpatient.

The map visualization may be provided to the physician or otherassociated with the physician in real-time. For example, once thebiological data associated with the new patient is provided to theanalysis server, the analysis server may provide the map visualizationback to the physician or other associated with the physician within areasonably short time (e.g., within seconds or minutes). In someembodiments, the physician may receive the map visualization over anytime.

The map visualization may be provided to the physician in any number ofways. For example, the physician may receive the map visualization overany digital device such as, but not limited to, an office computer,IPad, tablet device, media device, smartphone, e-reader, or laptop.

In step 1712, the physician may assess possible different clinicaloutcomes based on the map visualization. In one example, the map-aidedphysician may make decisions on therapy and treatments depending onwhere the patient lands on the visualization (e.g., survivor ornon-survivor). The map visualization may include annotations or labelsthat identify one or more sets of groupings and interconnections asbeing associated with one or more clinical outcomes. The physician mayassess possible clinical outcomes based on the position(s) on the mapassociated with the new patient.

FIG. 18 is a block diagram of an exemplary digital device 1800. Thedigital device 1800 comprises a processor 1802, a memory system 1804, astorage system 1806, a communication network interface 1808, an I/Ointerface 1810, and a display interface 1812 communicatively coupled toa bus 1814. The processor 1802 may be configured to execute executableinstructions (e.g., programs). In some embodiments, the processor 1802comprises circuitry or any processor capable of processing theexecutable instructions.

The memory system 1804 is any memory configured to store data. Someexamples of the memory system 1804 are storage devices, such as RAM orROM. The memory system 1804 can comprise the ram cache. In variousembodiments, data is stored within the memory system 1804. The datawithin the memory system 1804 may be cleared or ultimately transferredto the storage system 1806.

The storage system 1806 is any storage configured to retrieve and storedata. Some examples of the storage system 1806 are flash drives, harddrives, optical drives, and/or magnetic tape. In some embodiments, thedigital device 1800 includes a memory system 1804 in the form of RAM anda storage system 1806 in the form of flash data. Both the memory system1804 and the storage system 1806 comprise computer readable media whichmay store instructions or programs that are executable by a computerprocessor including the processor 1802.

The communication network interface (com. network interface) 1808 can becoupled to a data network (e.g., communication network 204) via the link1816. The communication network interface 1808 may support communicationover an Ethernet connection, a serial connection, a parallel connection,or an ATA connection, for example. The communication network interface1808 may also support wireless communication (e.g., 1802.11a/b/g/n,WiMAX). It will be apparent to those skilled in the art that thecommunication network interface 1808 can support many wired and wirelessstandards.

The optional input/output (I/O) interface 1810 is any device thatreceives input from the user and output data. The optional displayinterface 1812 is any device that may be configured to output graphicsand data to a display. In one example, the display interface 1812 is agraphics adapter.

It will be appreciated that the hardware elements of the digital device1800 are not limited to those depicted in FIG. 18. A digital device 1800may comprise more or less hardware elements than those depicted.Further, hardware elements may share functionality and still be withinvarious embodiments described herein. In one example, encoding and/ordecoding may be performed by the processor 1802 and/or a co-processorlocated on a GPU.

FIG. 19 shows example a landmark module 1900 configured to identifylandmark points that approximate or represent a larger collection ofdata points in accordance with various embodiments. In this example,landmark module 1900 comprises landmark selection module 1902, adistance calculation module 1904, a landmark distance identificationmodule 1906, a landmark distance storage module 1908, a landmarkdistance comparison module 1910, and a landmark assignment module 1912.

The landmark selection module 1902 may be configured to randomly selecta first subset of the data points to assign as an initial set oflandmark points. For example, the landmark selection module 1902 mayselect an initial set of points from the finite metric space as alandmark set L. It will be appreciated that the landmark selectionmodule 1902 may select points pseudo-randomly (e.g., randomly within thebounds of software or computer implementation) and/or in combinationwith other methods (e.g., randomly within portions of the finite metricspace or based, in part, on density of information). Landmark selectionmodule 1902 may select points in any number of ways (e.g., the landmarkselection module 1902 may select points based on any methodology and/ormay not select points randomly).

The distance calculation module 1904 may be configured to calculate thedistances between a respective non-landmark data point and each landmarkpoint in the finite reference space. In some embodiments, the distancecalculation module 1904 stores some or all of the information for lateruse.

The landmark distance identification module 1906 may be configured toidentify the shortest distance from among the distances between therespective non-landmark data point and each landmark. The shortestdistance between a non-landmark data point and a landmark data point mayindicate the closest landmark to that particular non-landmark datapoint.

The landmark distance storage module 1908 may be configured to store theshortest data point distance for the respective non-landmark data pointas a landmark distance for that data point. The landmark distancecomparison module 1910 may be configured to determine a longest landmarkdistance from among the shortest distances (e.g., stored by the landmarkdistance storage module 1908) to a nearest landmark for each data point.

The landmark assignment module 1912 may be configured to add a datapoint associated with the longest landmark distance to the initial setof landmark points thereby adding a new landmark and creating a new setof landmark points.

As described herein, the landmarks (L) are a subset of the collectiondata points in the finite metric space. The landmarks may be chosen suchthat the subset is representative of or to approximate the receiveddata. In some embodiments, the landmarks are chosen to reflect both the“average” and “extreme” behavior of the data points in the space and,thus, analytics and other operations performed on the landmark set as anapproximation of the behavior of the whole metric space (X). In someembodiments, the landmark points may be used as a means of increasingscale and performance when working with a large collection of data byonly operating on a subset of a space.

FIG. 20 is a flow chart 2000 depicting an example method for generatinga set of landmark points from a data set in some embodiments. Thefollowing discussion regarding the steps in FIG. 20 will be describedwith references to FIGS. 21A-D and FIG. 22A-C. In step 2002, thelandmark selection module 1902 receives a set of data points defining afinite metric space. For example, receiving data may include landmarkselection module 1902 accessing a data structure containing a very largevolume of multidimensional data, as shown in FIG. 21A.

FIG. 21A shows example metric space 2100 containing data in accordancewith various embodiments. Since the amount of data shown in metric space2100 handled by the methods and algorithms discussed herein may be large(e.g., on the order of 200 million+ data points), subset 2102 of metricspace 2100 will be used for discussion purposes. Accordingly, FIG. 21Bshows subset 2102 composed of individual data points 2104 in accordancewith some embodiments.

At step 2004, the analysis system 2606 selects a random subset ofindividual data points 2104 as a first set (e.g., an initial set) oflandmark points. To illustrate this step, FIG. 21C shows example randomlandmarks R₁, R₂, R₃, and R₄ that have been randomly selected as initiallandmarks. Since metric space 2100 is large (e.g., 200 million+ datapoints), points selected at random tend to be located in high densityareas, which is a benefit when attempting to choose a subset of pointsthat represent the characteristics of the larger space. For example, fora metric space of approximately 200 million data points, the number ofrandomly selected landmark points could be approximately 5,000 points.Thus, the probability that a significant portion of the randomlyselected landmarks may end up being outliers, for example, may be quitelow and the randomly selected landmarks end up being located in higherdensity data point regions.

At step 2006, for each non-landmark point, the distance calculationmodule 1904 calculates distances between that particular non-landmarkpoint and each landmark point. As used herein, the distances betweenlandmark points and individual data points 2104 are referred to as datapoint distances. Accordingly, FIG. 21D shows lines corresponding to datapoint distances to each landmark for three points (P₁, P₂, and P₃). Itshould be appreciated that, in various embodiments, the data pointdistances for all other points other than P₁, P₂, and P₃ and thelandmarks are also calculated, but of clarity and illustrative purposes,the lines shown in FIG. 21D have only been drawn for P₁, P₂, and P₃.Accordingly, in this example, each distance between P₁ and R₁, R₂, R₃,and R₄ is calculated, each distance between P₂ and R₁, R₂, R₃, and R₄ iscalculated, etc. until the distances between each non-landmark point andall the landmarks are calculated. FIGS. 22A and 22B show this process inmore detail.

FIG. 22A shows example data point distances between point P₁ and randomlandmarks R₁, R₂, R₃, and R₄. In this example, distance d₁ between P₁and R₁ is 3, distance d₂ between P₁ and R₂ is 5, distance d₃ between P₁and R₃ is 7, and distance d₄ between P₁ and R₄ is 6. In variousembodiments, the landmark distance for a respective non-landmark pointis defined as the shortest distance to its nearest landmark or theshortest data point distance. In this example, distances d₁, d₂, d₃, andd₄ are compared to each other to determine which is the shortestdistance to a landmark from P₁. In this example, distance d₁, between P₁and R₁, is the shortest distance and, thus, defined as landmark distance2202 for P₁. Accordingly, R₁ is the closest landmark to P₁ withcorresponding landmark distance 2202 (i.e., d₁=3).

Similarly, FIG. 22B shows example distances between point P₂ and randomlandmarks R₁, R₂, R₃, and R₄. In this example, distance d₅ between P₂and R₁ is 5, distance d₆ between P₂ and R₂ is 5, distance d₇ between P₂and R₃ is 9, and distance d₈ between P₂ and R₄ is 8. As above, distancesd₅, d₆, d₇, and d₈ are compared to each other to determine which is theshortest distance to P₂'s nearest landmark, which is distance d₅.Accordingly, distance d₅ between P₂ and R₁ is landmark distance 2204.Thus, R₁ is also the closest landmark to P₂ at landmark distance 2204(i.e., d₅=5), in this example.

Accordingly, the distance calculations described in FIGS. 22A and 22Bare, thus, calculated for P₃ and every other non-landmark point inmetric space 2100 and the distance calculations may be stored. Forexample, FIG. 22C shows an example table 2250 wherein distances for eachpoint are stored. Although FIG. 22C depicts a table, it will beappreciated that any data structure(s) or combination of datastructure(s) may be utilized. Further, although table 2250 includes alldistances from P1 to each landmark, it will be appreciated that, in someembodiments, a subset of the distances may be stored. In one example,only the shortest distance between P1 and the closest landmark may bestored.

Further, in this example, only the distances for points P₁ and P₂ areshown, but it should be appreciated that such a table or array wouldinclude distances for each non-landmark point. Thus, in one embodiment,table 2250 stores the distances for each point to each landmark inmetric space 2100. From these distances, a landmark distance (e.g.,shortest distance to a nearest landmark) for each point may beidentified and compared to generate a second set of landmark points.This process is discussed further with respect to FIGS. 23A-23D.

At step 2008, landmark distance identification module 1906 identifiesthe shortest data point distance from among the data point distances.FIG. 23A shows example landmark distances for points P₁, P₂, and P₃ tolandmark R₁ which can be used to demonstrate the selection of additionallandmark points. For example, landmark distance identification module1906 determines for each point which landmark point is the closestlandmark point for that respective point. This may include, for example,comparing the distance values d_(n) from table 2250 for each point todetermine which distance d_(n) is the shortest. Accordingly, in thisexample, the shortest between a landmark and P₁ is 3 (i.e., between P₁and landmark point R₁) and the shortest distance to a landmark pointfrom P₂ is 5 which is also to landmark point R₁.

Such an operation may use an indexable state for X (i.e., points such asP₁, P₂, and P₃ in metric space 2100), an indexable array for L (e.g.,L[l] is the index in X of the l'th landmark) where each random landmarkpoint R_(n) and subsequently determined landmark point is in L, anddClosest[x] which records the shortest distance between X[x] (i.e., P₁,P₂, P₃, etc.) and a respective closest landmark point, and inL[ ] withis true if x is in L.

At step 2010, landmark distance storage module 1908 stores the shortestdistance from each non-landmark point to a landmark point (or thedistance to the nearest landmark) in an array. FIG. 23B shows exampleshortest distances from each non-landmark point to each landmark point.In FIG. 23B, table 2350 contains the shortest distances between eachdata point P₁, P₂, and P₃ and its closest landmark, respectively.

In various embodiments, for each non-landmark point, the closestlandmark point is identified. As a result, a list of non-landmark pointsthat identify the same landmark point as the closest landmark point maybe identified. For example, for each such landmark point, a table suchas table 2350 may be generated that identifies the non-landmark pointsthat identify the same particular landmark point as being closest. Thetable 2350 may further identify distances between those non-landmarkpoints and the same particular landmark point. In this example, table2350 may contain the shortest distances between data points P₁, P₂, andP₃ and landmark point R₁. data point and only one landmark R₁.

At step 2012, landmark distance comparison module 1910 determines alongest landmark distance from among each of the shortest data pointdistances (or a longest landmark distance) from among each of thelandmark distances. For example, returning to FIG. 23A, random landmarkpoint R₁ is the landmark nearest to points P₁, P₂, and P₃ and, thus, thelandmark distance l_(n) (i.e., the distance to a nearest landmark) foreach of these points is its respective distance to R₁, which may bestored in table 2350. Thus, in this example, the landmark distance forP₁ is l₁=3, the landmark distance for P₂ is l₂=5, and the landmarkdistance for P₃ is l₃=4. Accordingly, landmark distance comparisonmodule 1910 compares these distances to identify the longest distancewhich, in this example, is l₂=5 shown circled in FIG. 23B, belonging topoint P₂.

Thus, with the longest landmark distance, P₂ is maximally far away fromthe random landmarks relative to the other non-landmark points and, atstep 2014, landmark assignment module 1912 adds P₂ to the set of randomlandmark points (or seed landmarks) to generate a new set of landmarkpoints. Thus, there is an initial set of randomly selected landmarkpoints (R) and max-min landmark points (MM) calculated along the way aresubsequently added to R to generate a set of landmarks (L). Accordingly,FIG. 23C shows point P₂ as new MM landmark point L₁.

In various embodiments, this process may start over to identify and adda second most maximally far away point to the set of landmark pointsafter L₁ has been added to the initial set of randomly selected landmarkpoints (R). Thus, steps 2002 to 2014 can be repeated with L₁ includedinto the set of landmark points (L) when determining the landmarkdistances for each point. Accordingly, FIG. 23D shows subset 2102 withL₁ as a new landmark where the distances between various points havebeen calculated. In this example, R₁ is no longer the closest landmarkto points P₁ and P₃ with the inclusion of L₁ and L₂. For example, P₁ isnow a distance d_(1′)=2 from its nearest landmark L₁ and P₃, whosenearest landmark is also L₁, is now a distance d_(3′)=2 from L₁.Further, as shown in FIG. 23D, the distance d_(4′)=3 between point P₄and R₁ and the distance d_(5′)=4 between point P₄ and newly added MMlandmark point L₂ since d_(5′) is larger than d_(4′), d_(3′), andd_(1′).

In one example, a method for generating a set of landmark points canutilize a process called PROCESS_x_AND_l(X,l), for example, thatdetermines the distances between each point and each landmark point,identifies the closest landmark for each point (dClosest[ ]), andupdates an array of dClosest[ ] for each point. Subsequently, a processcalled FIND_NEXT_L(l) can add a new MM landmark at 1 to the set oflandmarks (L). For example, PROCESS_x_AND_l(x,l) can be implemented asfollows:

double dist=distance(x, L[l]);

if (dist<dClosest[x]) dClosest[x]=dist;

FIND_NEXT_L(l) can be implemented as follows:

double closestD = -Double.MAX_VALUE; for (int x = 0; x < |X|; x++) {  if(!inL[x] && (dClosest[x] > closestD)) { closestD = dClosest[x]; L[l] =x;

Thus, referring back to FIG. 23D, the method for generating a set oflandmark points can proceed by first selecting random landmarks R₁, R₂,R₃, and R₄ and, thereafter, successively calling PROCESS_x_AND_l(x,l)for each point in metric space 2100 (e.g., each x in X on every l in L).Accordingly, a first portion of a method for generating a set oflandmark points can be implemented as follows:

for l = 0, l < |R| l++ do  for x = 0, x < |X|, x++  doPROCESS_x_AND_l(x,l)

Once the first portion is completed, the remaining landmark points canbe looped over one at a time to find the next MM landmark in a secondportion of the method:

 for l = |R|, l < |L|, l++  do  FIND_NEXT_L(l) for x = 0, x < |X|, x++ do PROCESS_x_AND_l(x,l) done

If the landmark selection process is improperly implemented, it can beinefficient for large spaces. For example, the |L|x|X| matrix can behuge and, if the distance calculations are not ordered properly, thecomputation can page wildly. For example, as described above, thelandmark selection process iterates |L| (i.e., the number of landmarkpoints) times over the data X (i.e., the number of data points) ofmetric space 2100. If the data space X does not fit into availablememory on a computer system, the data in X gets read repeatedly fromdisc, with slow results.

It will be appreciated that landmarks may be used instead of an entiredata set for analysis. The landmark set may approximate the behavior ofa larger data set thereby allowing analysis of the landmark set forcomputational efficiency and speed.

The landmark process may be used at many different stages in topologicalanalysis (examples of topological analysis are described herein). Forexample, landmarks of data points mapped to a reference space may beidentified. The landmark set may then be utilized to create avisualization as also described herein. In one example, as discussedregarding FIG. 8, the input module 314 may receive data (e.g., data S).In one example, a user identifies a data structure and then identifiesID and data fields. Data S may be based on the information within the IDand data fields. It will be appreciated that data S may be a finitemetric space, or a generalization thereof, such as a graph or weightedgraph.

The input module 314 may generate reference space R. In one example,reference space R may be a well-known metric space (e.g., such as thereal line). The reference space R may be defined by the user. Theanalysis module 320 may generate a map ref( ) from S into R. The mapref( ) from S into R may be called the “reference map.”

A landmark set of data points may be determined using methods describedherein. The landmark set of data points may be a subset of the datapoints mapped into the reference space. For example, a first subset ofthe data points in the map may be selected to generate an initial set oflandmarks. Each data point of the first subset may define a landmarkpoint.

As discussed herein, for each non-landmark data point, first data pointdistances between a respective non-landmark data point and each landmarkpoint of the initial set of landmarks may be calculated, a firstshortest data point distance from among the first data point distancesbetween the respective non-landmark data point and each landmark pointof the initial set of landmarks may be identified, and the firstshortest data point distance as a first landmark distance for therespective non-landmark data point may be stored. Subsequently, one or agroup (i.e., a predetermined number of) non-landmark data point(s) withlongest first landmark distance(s) in comparison with other firstlandmark distances of other non-landmark data points may be identified.The non-landmark data point(s) associated with the longest firstlandmark distance as a first landmark point may be added to the initialset of landmarks to generate an expanded set of landmark points.

The resolution module 318 may generate a cover of R based on theresolution received from the user (e.g., filter(s), intervals, andoverlap—see discussion regarding FIG. 7 for example). The cover of R maybe a finite collection of open sets (in the metric of R) such that everypoint in R lies in at least one of these sets.

Having computed, for each landmark point, which “cover tags” it isassigned to, for each cover element, C_(d), the points may beconstructed, whose tags included, as set S(d). This may mean that everylandmark point s is in S(d) for some d, but some landmark points maybelong to more than one such set. In some embodiments, there is,however, no requirement that each S(d) is non-empty, and it isfrequently the case that some of these sets are empty. In thenon-parallelized version of some embodiments, each landmark point x isprocessed in turn, and x is inserted into a hash-bucket for each j inref_tags(t) (that is, this may be how S(d) sets are computed).

The analysis module 320 may cluster each landmark S(d) based on themetric, filter, and the space S. In some embodiments, a dynamicsingle-linkage clustering algorithm may be used to partition S(d).

The visualization engine 322 may identify nodes which are associatedwith a subset of the partition elements of all of the landmark S(d) forgenerating a visualization. Of the sets of points that are used, twonodes intersect provided that the associated node sets have a non-emptyintersection.

The visualization engine 322 may join clusters to identify edges (e.g.,connecting lines between nodes). Once the nodes are constructed, theintersections (e.g., edges) may be computed “all at once,” by computing,for each point, the set of node sets (not ref_tags, this time). That is,for each landmark s in S, node_id_set(s) may be computed, which is anint[ ]. In some embodiments, if the cover is well behaved, then thisoperation is linear in the size of the set S, and we then iterate overeach pair in node_id_set(s). There may be an edge between two node_id'sif they both belong to the same node_id_set( ) value, and the number oflandmark points in the intersection is precisely the number of differentnode_id sets in which that pair is seen. This means that, except for theclustering step (which is often quadratic in the size of the sets S(d),but whose size may be controlled by the choice of cover), all of theother steps in the graph construction algorithm may be linear in thesize of S, and may be computed quite efficiently.

The visualization engine 322 may generate the visualization ofinterconnected nodes.

The landmark process may be used at other stages in topologicalanalysis. For example, nodes may be determined based on complex datausing topological data analysis as described herein. The nodes may alsobe landmarked and a visualization may be generated that includes thenodes of landmark points. This subset of nodes may have in a mannersimilar to the larger set of all nodes.

In one example, as discussed regarding FIG. 8, the input module 314 mayreceive data (e.g., data S). In one example, a user identifies a datastructure and then identifies ID and data fields. Data S may be based onthe information within the ID and data fields. The input module 314 maygenerate reference space R. In one example, reference space R may be awell-known metric space (e.g., such as the real line). The analysismodule 320 may generate a map ref( ) from S into R. The map ref( ) fromS into R may be called the “reference map.”

The resolution module 318 may generate a cover of R based on theresolution received from the user (e.g., filter(s), intervals, andoverlap—see discussion regarding FIG. 7 for example). The cover of R maybe a finite collection of open sets (in the metric of R) such that everypoint in R lies in at least one of these sets.

The analysis module 320 may cluster each data point S(d) based on themetric, filter, and the space S. In some embodiments, a dynamicsingle-linkage clustering algorithm may be used to partition S(d).

The visualization engine 322 may identify nodes which are associatedwith a subset of the partition elements of all of the data points S(d)for generating a visualization. Of the sets of points that are used, twonodes intersect provided that the associated node sets have a non-emptyintersection.

The nodes may be landmarked. For example, an initial set of nodes may beidentified as landmark nodes. For each non-landmark node, first datapoint distances between a respective non-landmark node and each landmarknode of the initial set of landmarks may be calculated, a first shortestdata point distance from among the first data point distances betweenthe respective non-landmark node and each landmark node of the initialset of landmarks may be identified, and the first shortest data pointdistance as a first landmark distance for the respective non-landmarknode may be stored. Subsequently, one or a group (i.e., a predeterminednumber of) non-landmark data point(s) with longest first landmarkdistance(s) in comparison with other first landmark distances of othernon-landmark nodes may be identified. The non-landmark node(s)associated with the longest first landmark distance as a first landmarknode may be added to the initial set of landmarks to generate anexpanded set of landmark nodes.

The visualization engine 322 may join clusters to identify edges (e.g.,connecting lines between nodes). Once the nodes are constructed, theintersections (e.g., edges) may be computed “all at once,” by computing,for each point, the set of node sets. The visualization engine 322 maygenerate the visualization of interconnected nodes.

FIG. 24A shows an example wherein data in X does not fit into localmemory 2402 (e.g., Random Access Memory (RAM)) and is, therefore, readoff of long term storage 2404. In this example, landmark set 2408represents storage of the set of all landmark points and data point sets2406 a, 2406 b, and 2406 c represent three different portions of thedata space X (e.g., each of data points sets 2406 a, 2406 b, and 2406 ccontaining different data). Accordingly, in this example, landmark set2408 and only a first set 2406 a of data space X can fit in local memory2402.

In the example discussed herein, landmark set 2408 could represent anamount of data points on the order of about 5,000 points and data pointsets 2406 a, 2406 b, and 2406 c could represent an amount of data pointson the order of about 100 million+ data points. Thus, once data pointset 2406 a has been compared to landmark set 2408 to determine thedistance calculations, data point set 2406 a must be removed from localmemory 2402 to make room for data point set 2406 b. After removal ofdata point set 2406 a from local memory 2402, data point set 2406 b isread off disk 2404 and loaded into local memory 2402. Accordingly, oncedata point set 2406 b has been compared to landmark set 2408 todetermine those distance calculations, data point set 2406 b is removedfrom local memory 2402 and data point set 2406 c is read off disk 2404and loaded into local memory 2402. The process of reading this much dataoff of disk 2404 creates significant latency.

Since the number of landmark points does not change until after a newlandmark has been determined and added (i.e., after each iteration), thenumber of landmark points is effectively limited for each round ofdistance calculations and, thus, PROCESS_x_AND_l( ) may only depend on x(e.g., data point sets 2406 a, 2406 b, and 2406 c). Therefore,PROCESS_x_AND_l( ) can be called in any order on x and the set L values(Landmark point values), provided that the process is being called onall landmark (landmark set 2408) and non-landmark point pairs (e.g.,data point sets 2406 a, 2406 b, and 2406 c). As a result, the firstprocess described above may be reordered to process all landmark points(e.g., landmark set 2408) for each x (e.g., data point sets 2406 a, 2406b, and 2406 c) instead of all points in X for each landmark point L.Accordingly, FIG. 24B shows an example wherein data point sets 2406 a,2406 b, and 2406 c are stored in local memory 2402 instead of landmarkset 2408 in accordance with various embodiments. Thus, instead ofrepeatedly reading the large amount of data associated with data pointsets 2406 a, 2406 b, and 2406 c off of disk 2404, the comparatively muchsmaller amount of data associated with landmark set 2408 is read offdisk 2404. Thus, the first portion of a method for generating a set oflandmark points may be reordered (STEP1A) and implemented as follows:

 for x = 0, x < |X|, x++ do  for 1 = 0, l < |R|, l++ do PROCESS_x_AND_l(x,l) done

Since PROCESS_x_AND_l( ) only depends on x, a current state or snapshotof the set of landmarks can be stored in local memory 2402 andPROCESS_x_AND_l( ) can be altered to use that state when performing anext iteration of distance calculations. Accordingly, if that state fitsinto local memory 2402 along with, for example, J rows of X, and thereordered first portion (STEP1A) of the method for generating the set oflandmarks and be run with only |X|/J page faults. For example, since thenumber of landmark points L is generally much smaller than the set ofdata points |X|, the number of page faults when scanning X with L inlocal memory 2402 is approximately the same as the number of page faultswhen scanning X without L. For example, if M is a number of rows of Xwhich can be simultaneously stored in local memory 2402, then the numberof page faults associated with the first potion of the method (STEP1)before reordering is approximately |L|*|X|/M and the number of pagefaults associated with the first portion of the method after reordering(STEP1A) is approximately |X|/(M−|L|).

Further, given T threads, the data points of X can be split into stripessuch that at least T of these stripes can fit into local memory 2402along with L. Accordingly, each thread of T can independently process astripe, such that there are ‘T versions’ of STEP1A concurrentlyoperating. As the x values are partitioned, contention is minimal and wesee in practice speedups of a factor of T. Concurrent operations do notalways finish precisely at the same time, thus, in one example, eachthread may include spin-locks to acquire new a stripe in order. This canalso enable the stripes to be fairly small and kept roughly together asX is iterated over.

The max-min landmark selection process for T threads is somewhatdifferent, but it can be understood as a FIND_NEXT_L(l) followed by aSTEP1 with only one landmark (which is equivalent to STEP1A, in thiscase) instead of a STEP2. This means that the end of each STEP1A threadcan be synchronized to then run a FIND_NEXT_L( ) and then partition Xinto stripes and run the STEP1A piece in parallel. As FIND_NEXT_L( )iterates over two (or more) arrays (e.g., one of booleans and another ofdoubles or floats), it may have paging issues only for truly giganticspaces or machines with small amounts of memory.

In various embodiments, instead of determining a single new landmarkpoint for each iteration of the aforementioned method of generating aset of landmark points, multiple landmark points can be chosen at atime. FIGS. 25A-25C show a process for generating a set of landmarkpoints wherein multiple landmark points are selected for each iterationof distance calculations described above. In at least one embodiment,for each iteration of distance calculations, the top “n” data pointsassociated with the longest distances to a nearest landmark point couldbe selected. The number “n” could vary, such as with the size of dataspace X, or it could be fixed to select a top predetermined number(e.g., 5) of the most distant data points, for example, from arespective landmark point for each iteration of distance calculations.

FIG. 25A shows subset 2102 with distances shown for points P₁, P₂, P₃,and P₄ to their respective closest random landmark (R₁, R₂, R₃, R₄). Inthis example, distance d₁ between P₁ and R₁ is 3, distance d₂ between P₁and R₂ is 5, distance d₃ between P₃ and R₃ is 7, and distance d₄ betweenP₄ and R₂ is 4 and these distances are shown in table 2550 of FIG. 25B.FIG. 25B shows example shortest distances from each non-landmark pointto each landmark point.

In this example, points P₂, P₃, and P₄ are in a top “n” data pointsbeing selected for this iteration based on each of their correspondingdistances to their nearest landmark point. For example, among an “n”number of landmark points being selected for this particular iteration,the distance d₁=3, between P₁ and R₁, is too short relative to otherdata points in subset 2102 and may not, therefore, be chosen forinclusion in the set of landmark points. Points P₂, P₃, and P₄, however,are chosen for inclusion in the set of landmark points with randomlandmark points (R₁, R₂, R₃, R₄).

Accordingly, FIG. 25C shows points P₂, P₃, and P₄ as landmarks L₁, L₂,and L₃ in this example. As can be seen in FIG. 25C, L₁ and L₂ are closetogether since they were selected without taking their relativedistances to each other into consideration and at least one of themwould not have been chose as a landmark point if only one landmark werechosen at a time. However, this process may work as an approximation andthe landmark points may not necessarily need to be perfectly spaced whenthe collection of data points is large. One way to potentially avoidchoosing landmarks that are too close to each other is by firstselecting a single landmark in a first iteration, a few such as 5landmarks in a second iteration, a single landmark again in a thirditeration, and so on. However, even if a few landmark points end upbeing close to each other, when taken into account with all otherlandmarks, the space can still be effectively approximated.

In some embodiments, more than one landmark point can be selected at atime by executing STEP1A on all landmarks at once, further resulting infewer iterations over X. In this example, identifying multiple MMlandmarks at a time may be accomplished by noticing that values ofdClosest[ ] decrease as more landmark points are added. Thus, the valuesof dClosest[x] may only stay the same or go down as more data points areadded to the set of landmark points. The landmark at l is, thus, the xin dClosest[ ] at step l−1 which has the largest value. As a result, ifx is the MM landmark at l, an obvious candidate for the MM landmark atl+1 may be the x′ which has the second largest value in dClosest[ ] atl−1. In one example, if dClosest[x′] does not decrease, x′ will be thelandmark point chosen at l+1 using any of the aforementioned processesfor selecting landmark points. In other words, if x′ is further from xthan from the closest of the previous l−1 landmark points, then it maybe the l+1th landmark. This pruning can be extended by remembering somefixed number K of largest indices and values for dClosest[ ], and thenpruning these by various heuristic processes. For instance, STEP2 fromabove can be altered as the following:

double dist = distance(x, L[l]); if (dist < dClosest[x]) {  dClosest[x]= dist;  insertKLargest(x, dist);

In this example, insertKLargest( ) maintains a data structure whichrecalls the K-largest pairs (x, distance). We can then iterate over theK pairs, largest first, to recompute the dClosest[ ] values by addingthe point with associated with the largest distance to the set oflandmark points. Any values which remain larger than other dClosest[ ]values can be considered reliable and values which remain as the processcontinues to add additional points to the set of landmark points as thevalues of dClosest[ ] are adjusted along the way are themselves thelandmark points this process is searching for. This process might failto find any additional landmark points, however, as all the K-largestpairs might be part of a cluster eliminated by a newest landmark in theprocess. In practice, however, this process generally results inadditional landmark points, and can reduce the number of iterations overX by the average number of landmarks generated.

In one example, the distance calculations can use only the smallestvalues of the distances for a given data point x, such that the numbervaries depending on the K in STEP2. The following method (STEP2A) isequivalent, and for certain spaces, can be more efficient:

double dist = distanceUpToLimit(x, L[l], KLargest(x));  if (dist <dClosest[x]) { dClosest[x] = dist; insertKLargest(x, dist);

In this example, distanceUpToLimit( ) will quit calculating the distancewhen it is known that the distance may be “too large to be interesting.”Since many points can be considered relatively far away, and spaces ofdimensionality in the millions are not uncommon (and those in thethousands and tens of thousands are routine), this can lead tosignificant performance improvements.

In another example, the distances within a stripe can be computed in asingle pass where the computation for each pairwise distances areinterleaved, rather than computed serially. Such a process can beutilized at a low level and useful when using a smaller number ofthreads and metrics for which the distances can make use of specializedvectorization hardware. This approach has the potential to deliverimproved performance, as it eliminates a lot of the redundantcomputations introduced when each distance in a stripe is computedserially. Testing, however, indicates that under load, the performanceadvantage associated with interleaving is marginalized, as threads spendan increasing amount of time waiting for the memory subsystem torespond.

Accordingly, interleaving, in effect, may do a kind of loop unrolling atthe lowest level of the metric calculation. For example:

double l2(double *in0, double *in1, int len) {  double accum = 0;  for(int i = 0; i < len; i++) {  accum += (in0[i] − in1[i]) * (in0[i] −in1[i]);  }  return sqrt(accum); } void interleaved_l2(double *x0,double *x1, double *x2, double *x3, double *y, int len, double *accum) { double accum0 = 0, accum1 = 0, accum2 = 0, accum3 = 0;  for (i = 0; i <len; i++) { double yval = y[i]; accum0 += (x0[i] − yval) * (x0[i] −yval); accum1 += (x1[i] − yval) * (x1[i] − yval); accum2 += (x2[i] −yval) * (x2[i] − yval); accum3 += (x3[i] − yval) * (x3[i] − yval);  } accum[0] = sqrt(accum0);  accum[1] = sqrt(accum1);  accum[2] =sqrt(accum2);  accum[3] = sqrt(accum3);

In this example, “yval” does not need to be reloaded and the process wasable to avoid doing three of the four loop checks. The larger “len” isthe more this will matter. Thus, in this example, the distances may allbe computed in a single pass where the computation of each pairwisedistance was interleaved, rather than computed serially. Accordingly,this approach eliminates some redundant computations introduced wheneach distance in a stripe is computed serially, thereby, increasingcomputational efficiency.

Systems and methods described herein may be utilized in big dataanalysis. Big data is a term that refers to data sets that are largeand/or complex such that traditional data processing of the prior artmay be inadequate or limited. Massive data sets may include hundreds ofthousands, millions, or even billions (or more) data points and/or anynumber of characteristics per data point. There may be significanthardware, service, and/or financial limitations that must be consideredwhen attempting to analyze large data sets (e.g., up to an includingmassive data sets as described above).

For example, system resource constraints, network limitations, serviceconstraints, and/or algorithmic limitations (within a larger analyticalframework) may impact analysis of large data sets. One or more of theselimitations (e.g., insufficient memory) may cause system failure beforelarge data sets are analyzed. Alternately, one or more of theselimitations may slow analysis to the point of impracticality. As aresult, when considering analyzing large data sets (including how alarge data set is to be analyzed), system resource constraints, networklimitations, service constraints, and/or algorithmic limitations.

A system resource is any physical or virtual component of limitedavailability within a computer system. Examples of physical systemresources that may have limitations of access and/or performanceinclude, but are not limited to, central processing units (CPUs), randomaccess memory (RAM), hard disk, cache space (e.g., CPU cache, MMUcache), network throughput, electrical power, input/output operations,and the like. Virtual system resources may include but are not limitedto files (e.g., file handles), network connections (e.g., networksockets), and memory. Resource management may include, but is notlimited to resource leaks (e.g., releasing a resource when a process hasfinished using it) and resource contention (when multiple processes wishto access a limited resource).

Network limitations include, but are not limited to, limitations ofbandwidth, capacity, performance, and/or the like used for transferringdata from one digital device to another. For example, when transferringall or parts of large data sets for analysis or during analysis, networklimitations may cause data transfer to fail or to be too slow to beimpractical. Network performance limitations may also impact systemresources. For example, even if data sets may be transferred across anetwork, if the performance is not sufficient, system resources may failor become unstable (e.g., receiving too much data or receiving data tooslowly).

Service constraints may include limitations on usage of servicesprovided by others that are used for data analysis. For example, cloudservers (e.g., servers available over the internet) provided by thirdparties may be used for data analysis. Service providers, however,typically charge for performance including quality of performance. As aresult, not only must server and performance limitations be consideredwhen selecting one or more service providers (e.g., and selecting one ormore servers of a service provider based on performance to perform allor part of the analysis), server and performance limitations of theseservers may be considered for potentially impacting analysis. Further,the costs of using servers may be considered in performing big dataanalysis. Costs associated with server and performance limitations mayinclude, but are not limited to, cloud constraints, server runninghours, storage costs, snapshot costs, read and write request costs,archiving costs, database running costs, database transaction costs(IO), and/or data transfer costs (including within a deployment andoutside of a deployment). Financial constraints may render analysisimpractical.

Algorithmic limitations may also be considered when determining how toanalysis large data sets. Often an issue in optimization (e.g., “Big-O”problems), algorithmic efficiency may relate to the time it takes forthe algorithm to run as a function of input size. Algorithmiclimitations may slow analysis of large data sets or stop analysiscompletely.

Multiple systems (e.g., local digital devices and/or digital devicesover a network) may be used on different parts of a large data setand/or perform portions of analysis. Each system, including constraintsin data transfer, may have limitations that impact performance orcapability. Similarly, multiple analytics programs and/or otherresources may be impacted by the performance of one or more systemresource constraints, network limitations, service constraints, and/oralgorithmic limitations. As a result, efficiencies and handling may beconsidered for speed and performance when planning to use digitaldevices, analytics programs, and/or other resources that work together.

Landmarking data sets, as described herein (particularly with thediscussion regarding FIGS. 19-25A-C), describe systems and methods forgenerating a landmark data set from an original data set. The landmarkdata set is generated to potentially include insights, information,and/or behavior of the larger original data set. Landmarking, asdescribed herein, may be used as a process for improving performanceand/or overcoming system resource constraints, network limitations,service constraints, and/or algorithmic limitations. For example,landmarking one or more large data sets of analysis may be used toincrease speed and/or enable analysis of large data sets when analyzingthe entire data set without landmarking is unwieldy, impractical, and/ornot possible. Landmarking may also enable more efficient use ofcomputational resources even if the entire data set may be analyzedwithout significant constraint.

It will be appreciated, however, that data sets may be so large thateven landmarking a large data set and/or analyzing such landmarks may beimpractical in view of system resource constraints, network limitations,service constraints, and/or algorithmic limitations. Some data sets maybe sufficiently massive such that computational resources may be unableto load data into memory, process the data, store the data, generate avisualization, and/or transfer data to identify all landmarks. Systemresource constraints, network limitations, service constraints, and/oralgorithmic limitations may be considered when selecting digital devicesto use, determining how landmarking is to be performed, and/ordetermining how to perform analysis using the landmarks.

In this example, this process has at least two scalability bottlenecksthat become problematic when operating on large data sets (e.g., datasets that contain billions of rows and/or are terabytes in size): 1. Ona very large set of points X computing the list of functions on X can becomputationally prohibitive. 2. Clustering points within each bucket canbe a problem if the number of points in the bucket is very large. Someembodiments describe example procedures to construct topologicalsummaries at scale.

In one example solution, a process of topological summary constructioncan be employed and/or modified with one or more of the followingobjectives in mind:

-   -   Prevent an explosion in computation cost as the size of the        dataset grows    -   Be amenable to division into a sub jobs that can more easily run        on commodity compute boxes with limited memory resources;        horizontally scalable out-of-core processing with reduced or        minimal inter-node communication or synchronization    -   Avoid expensive data shuffling across terabyte files—such as        required in step 3 of the original flow

FIG. 26 is an example environment 2600 in which embodiments may bepracticed. In various embodiments, landmarking, data analysis and/orgeneration of an interactive visualization may be performed locally(e.g., with software and/or hardware on a local digital device), acrossa network (e.g., via cloud computing), or a combination of both. In manyof these embodiments, a large data set may be distributed and/orlandmarked (e.g., identifying and/or determining landmark points in alarge data set and/or portions of a large data set discussed herein) byany number of computation devices (e.g., computation devices 2604 a-d).

Environment 2600 comprises a data source system 2602, computationdevices 2604 a-d, an analysis system 2606, and a communication network2608. Environment 2600 depicts an embodiment wherein functions areperformed across a network. In this example, the user(s) may takeadvantage of cloud computing and/or computation devices accessible overa network.

The data source system 2602, computation devices 2604 a-d, and theanalysis system 2606 may each be any digital device or any number ofdigital devices. A digital device is any device that includes memory anda processor. Digital devices are further described in FIG. 18. The datasource system 2602, computation devices 2604 a-d, and the analysissystem 2606 may be any kind of digital device that may be used toaccess, analyze and/or view data including, but not limited to a server,desktop computer, laptop, notebook, or other computing device.

Although only one device is depicted in FIG. 26 for each of the datasource system 2602, computation devices 2604 a-d, and the analysissystem 2606, it will be appreciated that there may be any number ofdevices. For example, the data source system 2602 may comprise a systemincluding any number of digital devices that are local or remote fromeach other (e.g., accessible over a network that may include or may notinclude communication network 2608). Similarly, each computation devices2604 a-d may include any number of digital devices that are local orremote from each other (e.g., accessible over a network that may includeor may not include communication network 2608). Still further, theanalysis system 2606 may include any number of digital devices that arelocal or remote from each other (e.g., accessible over a network thatmay include or may not include communication network 2608).

The data source system 2602 may provide all or part of a large data setto be analyzed. In some embodiments, the data source system 2602provides a part of the large data set to each of the computation devices2604 a-d and/or the analysis system 2606 for landmarking (e.g., aprocess to determine landmark points of all or a part of the data set).In this example, the data source system 2602 may provide one or moresubsets of the large data set to each computation device 2604 a-d and/orthe analysis system 2606. Each subset may or may not contain dataexclusive of other subsets of the large data set. In some embodiments,the data source system 2602 may perform landmarking and analysis of allor part of the data set (e.g., the data source system 2602 may be acomputation device 2604 a and/or an analysis system 2606).

It will be appreciated that the data source system 2602 may include anynumber of servers or devices that store all or part of a large data set.For example, the large data set may contain hundreds of thousands,millions, or billions of patient records to be analyzed. There may beany number of data source devices 2602 owned and/or operated by anynumber of health care service providers that may provide all or some ofthe data sets to be analyzed.

The data source system 2602 may provide and/or store all or part of thedata set. In various embodiments, the data source system 2602 storesdatabases and/or other data structures. In one example the data sourcesystem 2602 may be or include a secure server wherein a user may storedata over a secured connection (e.g., via https). The data may beencrypted and/or backed-up. In some embodiments, the data source system2602 is operated by a third-party such as AMAZON's S3 service.

The computation devices 2604 a-d may include any number of digitaldevices that performs landmarking and/or data analysis (e.g.,topological data analysis). For example, each computation device 2604a-d may select landmark points from a different subset of the originallarge data set. Landmarking of a data subset is described herein. Theremay be any number of computation devices 2604 a-d (e.g., greater or lessthan 4) to landmark and/or analyze any number of data subsets.

The communication network 2608 may be any network that allows digitaldevices to communicate. The communication network 2608 may be or includethe internet and/or include LAN and WANs. The communication network 204may support wireless and/or wired communication.

The analysis system 2606 may include any number of digital devicesconfigured to analyze data (e.g., all or part of the data set providedby the data source device(s) 2602). The analysis system 2606 may usetopological data analysis on any number of the subsets of the data set,the entire data set, and/or landmark information (e.g., expandedlandmark subsets of landmark points) received from the computationdevices 2604 a-d. Example functions of the analysis system 2606 arefurther described herein. The analysis server may be the analysis server208 described herein (e.g., see FIG. 2).

In various embodiments, the analysis system 2606 may perform manyfunctions to interpret, examine, analyze, and display data and/orrelationships within data. In some embodiments, the analysis system 2606performs, at least in part, topological analysis of one or more subsetslarge datasets applying metrics, filters, and resolution parameterschosen by the user.

The analysis system 2606 may generate graphs in memory, visualizedgraphs, and/or an interactive visualization of the output of theanalysis. As discussed herein, in some embodiments, an interactivevisualization allows the user to observe and/or explore relationships inthe data. In various embodiments, the interactive visualization allowsthe user to select nodes comprising data that has been clustered. Theuser may then access the underlying data, perform further analysis(e.g., statistical analysis) on the underlying data, and/or manuallyreorient the graph(s) (e.g., structures of nodes and edges describedherein) within the interactive visualization. The analysis system 2606may also allow for the user to interact with the data, see the graphicresult.

The graphs in memory and/or visualized graphs may also include nodes(e.g., graphical nodes or vertices) and/or edges as described herein.Graphs that are generated in memory may not be depicted to a user butrather may be generated in memory of a digital device. Visualized graphsare rendered graphs that may be depicted to the user.

In various embodiments, the analysis system 2606 may determine how todivide the data set to subsets. Further, the analysis system 2606 maydetermine a size for the initial subset of landmarks, the total numberof landmarks to be determined for each subset of landmark point, and/ora total number of landmarks to be used for analysis. Thesedeterminations may be based on system resource constraints, networklimitations, service constraints, and/or algorithmic limitations of anynumber of the data source system 2602, the computation devices 2604 a-d,the analysis system 2606, and/or the communication network 2608. Theseprocesses are further described herein.

FIG. 27 is a flowchart for determining landmark points using any numberof computation devices in some embodiments.

As previously discussed, identification of landmark points may be viewedat a high level as follows:

-   -   From the set of points X, the analysis system 2606 constructs a        subset of ‘representative’ points called the landmark set, L.        The landmark set of points has the following properties:        -   a. For most points x in X, which is not in L, there exists a            particular assigned landmark point in L, called L_x. For            example, one simple way the analysis system 2606 may arrive            at this assignment is by finding the landmark point which is            closest (using a similarity or dissimilarity measure such as            a proper distance metric) to x.        -   b. For some points x in X, there may be no assigned landmark            point. In some embodiments, the analysis system 2606 may            compare a distance between x and its closest landmark point            to an anomaly threshold. If the distance is greater or equal            to the anomaly threshold, then x may not be assigned a            landmark point. Points x in X that have no assigned landmark            may be categorized as “anomalies.” Indeed, it is possible to            envisage designing a separate workflow for detecting            anomalies; if a user is interested in anomaly detection, the            analysis system 2606 can be readily determined by returning            to the user those points that are not assigned a closest            landmark point.

In order to overcome slowness or constraints in computation, determininglandmark points may be performed by any number of computation devices(e.g., computation devices 2604 a-d). For example, it may be assumedthat, in some implementations, a single digital device may readilycompute the distance between any 2 points in the dataset—the entiredataset possibility being memory resident on a single device (e.g., datasource system 2602). However, the size of the dataset may be thousandsof times larger than the memory resources on any single digital device.

Accordingly, in this example approach, a digital device such as theanalysis system 2606 or the data source system 2602 may divide theoriginal data set X into a number of subsets. The subsets may be offixed size or variable size. The data set may be sub-divided in anynumber of ways including, but not limited to random or pseudo-random(without replacement). Is the data set may be sub-divided based on anycharacteristic (e.g., field or column of the data set) or combination ofcharacteristics).

These subsets, which may be as small as 100 MB each for example, may beeach landmarked in isolation on a different computation device 2604without consideration to the larger dataset. In some embodiments, thisphase of the computation is eminently scalable, as any number oflandmark subset computations may be executed in parallel if there aresufficient computational resources.

In some embodiments, once these “subset” landmarking operations arecomplete, if the total number of landmarks is still not tractable, theanalysis system 2606 may optionally iteratively landmark the chosenlandmark subsets until a manageable set of final landmarks are chosen.

In step 2702, the data source system 2602 may receive a set of datapoints (e.g., a data set). As discussed herein, the data source system2602 may include or communicate with any number of devices that storeall or part of the data set. The data set may be contained in any numberof data structures. In one example, the data set (e.g., the originaldata set) is shown in FIG. 21A.

In step 2704, the analysis system 2606 or the data source system 2602determines a division of data points among computation devices. In thefollowing discussion, the analysis system 2606 is discussed asperforming steps 2704-2708, however, it will be appreciated that thedata source system 2602 may perform any number or all of these steps.

In various embodiments, the analysis system 2606 may determine divisionof data points among computation devices based on hardwarecharacteristics of the computation device. For example, a computationdevice with greater capacity (e.g., network connectivity, memory,processing power, storage, and/or the like) may receive larger datapoints than other computation devices. In another example, the analysissystem 2606 may determine division of data points among computationdevices based on limitations of the computation devices, limitations ofthe network that allows communication between computation devices,limitations of the data source system 2602 to transfer the data, and/orthe analysis system 2606 capability to analyze data (e.g., landmarkpoints).

In various embodiments, the analysis system 2606 determines division ofdata points based on system resource constraints, network limitations,service constraints, and/or algorithmic limitations (within a largeranalytical framework) as discussed herein. For example, a systemresource is any physical or virtual component of limited availabilitywithin any number of computation devices 2604 a-d).

Network limitations include limitations of the communication network2608 including limitations on bandwidth, capacity, performance, and/orthe like in transferring data from one digital device to another.

Service constraints may include limitations on usage of servicesprovided by others that are used for data analysis. For example, cloudservers (e.g., servers available over the internet) may be utilized ascomputation devices. As a result, not only computation deviceperformance limitations be considered when selecting one or more serviceproviders (e.g., and selecting one or more computation devices of aservice provider based on performance), computation devices limitationsmay be considered for potentially impacting analysis. Further, the costsof using servers may be considered in performing big data analysis.Costs associated with computation device performance limitations mayinclude, but are not limited to, cloud constraints, server runninghours, storage costs, snapshot costs, read and write request costs,archiving costs, database running costs, database transaction costs(IO), and/or data transfer costs (including within a deployment andoutside of a deployment).

The analysis system 2606 may also consider algorithmic limitations indetermining division of the data set among the computation devices 2604a-d. The analysis system 2606 may take into account limitations,constraints, and performance of multiple systems (e.g., local digitaldevices and/or digital devices over a network) that may be used ondifferent parts of a large data set in determining division of the dataset

In various embodiments, the analysis system 2606 reviews any number ofthese constraints and limitations. In one example, the analysis system2606 identifies those constraints and limitations of the computationdevices 2604 a-d and/or the analysis system 2606 that will either stopanalysis and/or slow analysis below a desired threshold (e.g., fourhours). The analysis system 2606 may determine division of the data setto enable analysis and/or determine divisions of the data set in view ofspeed and efficiency considerations over the constraints and/orlimitations.

In step 2706, the analysis system 2606 determines a number of landmarkpoints for an analysis landmark set and a number of landmark points foreach subset of landmark points. The analysis landmark set may includeeach subset of landmark points. In some embodiments, the analysislandmark set includes landmark points selected from a union of thesubsets of landmark points.

As discussed herein, each computation device 2604 a-d may determinelandmark points for a subset of the data set thereby creating anexpanded landmark subset of landmark points. In one exampleimplementation, each computation device 2604 a-d may select an initialsubset of landmark points and then may add landmark points until anexpanded landmark subset of landmark points is obtained. The analysissystem 2606 may determine the size (e.g., number) of initial subset oflandmark points of each computation device 2604 a-d and/or the number oflandmark points for each expanded landmark subset of landmark points ofeach computation device 2604 a-d. The size of the initial subset oflandmark points may be the same or different for any number ofcomputation devices 2604 a-d. Similarly, the size of the expandedlandmark subset of landmark points may be the same or different for anynumber of computation devices 2604 a-d.

In various embodiments, the analysis system 2606 may determine thenumber of the initial set of landmark points and/or the expandedlandmark set of landmark points based on hardware characteristics of thecomputation device. For example, a computation device with greatercapacity (e.g., network connectivity, memory, processing power, storage,and/or the like) may determine a larger number of initial landmarkpoints in the initial set and/or a larger number of landmark points inthe expanded landmark set of landmark points of a larger number datapoints of a larger data subset than other computation devices. Inanother example, the analysis system 2606 may determine the number oflandmark points in the initial set of landmark points and/or theexpanded landmark set of landmark points based on system resourceconstraints, network limitations, service constraints, and/oralgorithmic limitations. For example, the analysis system 2606 maydetermine a number of landmark points for the initial set or expandedlandmark set based on limitations of the computation devices,limitations of the network that allows communication between computationdevices, limitations of the data source system 2602 to transfer thedata, and/or the analysis system 2606 capability to analyze data (e.g.,landmark points).

In another example, the analysis system 2606 may also consideralgorithmic limitations in determining the number of landmark points inthe initial set of landmark points and/or the expanded landmark set oflandmark points among the computation devices 2604 a-d. The analysissystem 2606 may take into account limitations, constraints, andperformance of multiple systems (e.g., local digital devices and/ordigital devices over a network) that may be used on different parts of alarge data set in determining division of the data set

It will be appreciated that that analysis system 2606 may determine thenumber of landmark points in the initial set of landmark points and/orthe expanded landmark set of landmark points based on both performancestrengths of different computation devices as well as limitations.

In various embodiments, the analysis system 2606 may determine a maximumnumber of data points the analysis system 2606 may analyze (e.g., amaximum number of an analysis landmark set and/or a maximum number ofdata points that can be added to nodes using the analysis landmark setas discussed herein). This determination may be based on system resourceconstraints of the analysis system 2606 (e.g., one or more digitaldevices of the analysis system 2606), network limitations (e.g.,limitations communicating information from the data source system 2602and/or the computation devices 2604 a-d between each other and/or theanalysis system 2606, or limitations of communicating informationbetween different digital devices of the analysis system 2606), serviceconstraints (e.g., one or more digital devices of the analysis system2606), and/or algorithmic limitations (within a larger analyticalframework of one or more digital devices of the analysis system 2606)).

In some embodiments, once a maximum number of data points of theanalysis system 2606 is determined, a size of subsets of landmark points(e.g., expanded landmark point subsets and/or initial subsets oflandmarks) may be determined for one or more computation devices 2604a-d. In one example, once the analysis system 2606 determines a maximumnumber of data points and/or landmark points that may be analyzed (e.g.,based on performance characteristics and/or constraints of the analysissystem 2606), then the analysis system 2606 may determine a size of theexpanded landmark subset of landmark points based on the maximum numberof data points and/or landmark points that may be analyzed (and/or basedperformance characteristics and/or based on limitations of thecomputation devices 2604 a-d). Further, in some embodiments, once theanalysis system 2606 determines size(s) of expanded landmark subset(s)of landmark points, then the analysis system 2606 may determine a subsetsize based on the expanded landmark subset of landmark points size(and/or based performance characteristics and/or based on limitations ofthe computation devices 2604 a-d). In some embodiments, once a maximumnumber of data points of the analysis system 2606 is determined, size ofsubsets of data points may be determined for one or more computationdevices 2604 a-d.

In step 2708, the analysis system 2606 divides and transfers data pointsof different subsets to different computation devices based ondetermined division of data points. In some embodiments, the analysissystem 2606 may divide and transfer data points to different computationdevices based on size of subsets of landmark points (e.g., expandedlandmark subset of landmark point and/or initial subsets of landmarks).

It will be appreciated that the analysis system 2606 may divide andtransfer the data points in any number of ways. For example, theanalysis system 2606 may provide commands to any number of data sourcedevices of data source system 2602 to provide any number of data pointsas subsets to any number of computation devices. In various embodiments,the analysis system 2606 may provide commands to one or more computationdevices 2604 a-d to start identifying landmark points for the initiallandmark set or the expanded landmark subset of landmark points if allor part of the subset is present on the computation device(s) 2604 a-d.

In various embodiments, the analysis system 2606 may provideinstructions and/or commands to any number of computation device(s) 2604a-d with instructions to set the size of the initial landmark set and/orthe size of the expanded landmark subset of landmark points.

At step 2710, each computation device 2604 a-d determines landmarkpoints and generates expanded landmark subset of landmark points. Forexample, each computation device 2604 a-d may receive a size of aninitial landmark point subset and/or a size of an expanded landmarksubset of landmark points. Each computation device 2604 a-d may select(e.g., randomly) a number of landmark points from their respective datasubset equal to the size of the initial landmark point subset. Eachcomputation device 2604 a-d may then add landmark points, respectively,to their particular initial landmark point subset to create respectiveexpanded landmark subset of landmark points. This process is furtherdiscussed with regard to FIG. 28.

At step 2712, each computation device 2604 a-d transfers theirrespective expanded landmark subset of landmark points to the analysissystem 2606. In some embodiments, each computation device 2604 a-d mayprovide all or some of the distances between each non-landmark point ofits particular data subset and each landmark point of the particularexpanded landmark subset of landmark points to the analysis system 2606.

In step 2714, the analysis system 2606 groups the f expanded landmarksubset of landmark points from the computation devices 2604 a-d tocreate a union of landmark subsets. Optionally, in step 2716, theanalysis system 2606 may determine an analysis landmark set of union oflandmark subsets by landmarking (e.g., finding landmark points of) theunion of landmark subsets. For example, the analysis system 2606 mayperform landmarking in a manner similar that performed by thecomputation devices in step 2710. In this example, the analysis system2606 treats the union of landmark subsets as a subset of data pointswithout any landmark points. The analysis system 2606 may select aninitial subset of landmark points from the data points of the union oflandmark subsets and then adds landmark points from the unselectedpoints of the union of landmark subsets to create an analysis landmarkset.

For example, the analysis system 2606 may determine an initial subset oflandmark points. The analysis system 2606 may randomly selected landmarkpoints (e.g., 5,000 points) from the union of landmark subsets. Theinitial landmark points may be selected in any number of ways. Selectionmay be random, may be “intelligently random” as discussed herein, bebased on any number of data characteristics of data in the data subsetor data set, and/or the like.

Subsequently, the analysis system 2606 may add additional landmarks inan intelligent fashion to reach a size for the analysis landmark set.For example, the analysis system 2606 may calculate data point distancesbetween a respective non-landmark data point and each landmark point ofthe initial subset of landmark points.

The analysis system 2606 may identify the shortest data point distancefrom among the data point distances. In various embodiments, for eachnon-landmark point, the closest landmark point is identified. Theanalysis system 2606 may determine a longest landmark distance fromamong each of the shortest data point distances (or a longest landmarkdistance) from among each of the landmark distances. The analysis system2606 may add the data point based on the longest landmark distance amongeach of the shortest data point distances to the analysis landmark set.The analysis system 2606 may add other data points in a similar fashionuntil a maximum size of the analysis landmark set is reached.

Alternately, in some embodiments, the analysis landmark set is the unionof landmark subsets.

In step 2718, the analysis system 2606, for each non-landmark point ofthe original data set (e.g., from the data source system 2602), theanalysis system 2606 may determine a location of a non-landmark datapoint relative to one or more of the closest landmarks of the expandedsubset of landmark points.

In various embodiments, the analysis system 2606 may determine whichnode (e.g., graphical node or vertex) associated with one or morelandmark points of the analysis landmark set of landmark points is to beassociated with a non-landmark point of the original data set. In someembodiments, the analysis system 2606 may perform all or part oftopological data analysis to identify nodes (e.g., graphical nodes orvertices) that include one or more landmark points of the analysislandmark set of landmark points. This process if further described withregard to FIG. 29.

The analysis system 2606 may then analyze each non-landmark point todetermine which node the non-landmark point should be a member. In someembodiments, the analysis system 2606 may determine one or more of theclosest landmark points from the analysis landmark set of landmarkpoints to a non-landmark point. The analysis system 2606 may determinewhich nodes are associated with the closest one or more landmark pointsand assign the non-landmark point to the one or more nodes based on thedetermination. Embodiments of the process are described with regard toFIGS. 30-32.

Although the analysis system 2606 is discussed as determining which nodeassociated with one or more landmark points of the analysis landmark setof landmark points is to be associated with a non-landmark point of theoriginal data set, it will be appreciated that the analysis system 2606may determine which node associated with one or more landmark points ofthe analysis landmark set of landmark points is to be associated with anon-landmark point of the union of landmark subsets (rather than theoriginal data set).

In step 2720, the analysis system 2606 may generate a visualization ofthe nodes (e.g., graphical nodes or vertices) and edges between thenodes. The generation of the visualization may be done in any number ofways. The process is further described herein. In one example, thevisualization may be generated in a manner similar to that discussedwith regard to FIGS. 3, 4, and 8.

FIG. 28 is a flowchart for a computation device 2604 a to create anexpanded landmark subset of landmark points in some embodiments. Invarious embodiments, there may be any number of computation devices 2604a-d creating any number of expanded landmark subsets of landmark points.The flowchart of FIG. 29 describes a process for one computation device2604 a, however, it will be appreciated that any number of computationdevices may each create a different expanded landmark subset of landmarkpoints based on a different subset of data points.

In step 2802, the computation device 2604 a selects a subset (e.g., agroup) of data points, each data point of the subset defining a landmarkpoint. The selected subset is a first sub-subset of landmarks. In someembodiments, the computation device 2604 a may determine an initialsubset of landmark points. The number of landmark points selected forthe initial subset of landmark points (e.g., the size of the initialsubset of landmark points) may be provided by the analysis system 2606.

For example, each computation device 2604 a-d may select a random subsetof individual data points as a first set (e.g., an initial set) oflandmark points. As discussed herein, to illustrate this step, FIG. 21Cshows example random landmarks R₁, R₂, R₃, and R₄ that have beenrandomly selected as initial landmarks. Since even a subset of datapoints may be large (e.g., million+ data points), points selected atrandom tend to be located in high density areas, which is a benefit whenattempting to choose a subset of points that represent thecharacteristics of the larger space.

One or more computation devices 2604 a-d may randomly selected landmarkpoints equal to the size provided by the analysis system 2606. Theinitial landmark points may be selected in any number of ways. Selectionmay be random, may be “intelligently random” as discussed herein, bebased on any number of data characteristics of data in the data subsetor data set, and/or the like.

Subsequently, each computation device may add additional landmarks in anintelligent fashion to reach an expanded landmark subset of landmarkpoints. In step 2804, the computation device 2604 a calculates datapoint distances between a respective non-landmark data point and eachlandmark point of the initial subset of landmark points. For example,for each non-landmark point, a computation device 2604 a may calculatedistances between that particular non-landmark point and each landmarkpoint of that computation device's particular initial landmark pointsubset. As used herein, the distances between landmark points andindividual data points 2104 are referred to as data point distances. Asdiscussed herein, FIG. 21D shows lines corresponding to data pointdistances to each landmark for three points (P₁, P₂, and P₃). It shouldbe appreciated that, in various embodiments, the data point distancesfor all other points other than P₁, P₂, and P₃ and the landmarks arealso calculated, but of clarity and illustrative purposes, the linesshown in FIG. 21D have only been drawn for P₁, P₂, and P₃. Accordingly,in this example, each distance between P₁ and R₁, R₂, R₃, and R₄ iscalculated, each distance between P₂ and R₁, R₂, R₃, and R₄ iscalculated, etc. until the distances between each non-landmark point andall the landmarks are calculated. FIGS. 22A and 22B show this process inmore detail.

FIG. 22A shows example data point distances between point P₁ and randomlandmarks R₁, R₂, R₃, and R₄. In this example, distance d₁ between P₁and R₁ is 3, distance d₂ between P₁ and R₂ is 5, distance d₃ between P₁and R₃ is 7, and distance d₄ between P₁ and R₄ is 6. In variousembodiments, the landmark distance for a respective non-landmark pointis defined as the shortest distance to its nearest landmark or theshortest data point distance. In this example, distances d₁, d₂, d₃, andd₄ are compared to each other to determine which is the shortestdistance to a landmark from P₁. In this example, distance d₁, between P₁and R₁, is the shortest distance and, thus, defined as landmark distance2202 for P₁. Accordingly, R₁ is the closest landmark to P₁ withcorresponding landmark distance 2202 (i.e., d₁=3).

Similarly, FIG. 22B shows example distances between point P₂ and randomlandmarks R₁, R₂, R₃, and R₄. In this example, distance d₅ between P₂and R₁ is 5, distance d₆ between P₂ and R₂ is 5, distance d₇ between P₂and R₃ is 9, and distance d₈ between P₂ and R₄ is 8. As above, distancesd₅, d₆, d₇, and d₈ are compared to each other to determine which is theshortest distance to P₂'s nearest landmark, which is distance d₅.Accordingly, distance d₅ between P₂ and R₁ is landmark distance 2204.Thus, R₁ is also the closest landmark to P₂ at landmark distance 2204(i.e., d₅=5), in this example.

Accordingly, the distance calculations described in FIGS. 22A and 22Bare, thus, calculated for P₃ and every other non-landmark point inmetric space 2100 and the distance calculations may be stored. Forexample, FIG. 22C shows an example table 2250 wherein distances for eachpoint are stored. Although FIG. 22C depicts a table, it will beappreciated that any data structure(s) or combination of datastructure(s) may be utilized. Further, although table 2250 includes alldistances from P1 to each landmark, it will be appreciated that, in someembodiments, a subset of the distances may be stored. In one example,only the shortest distance between P1 and the closest landmark may bestored.

Further, in this example, only the distances for points P₁ and P₂ areshown, but it should be appreciated that such a table or array wouldinclude distances for each non-landmark point. Thus, in one embodiment,table 2250 stores the distances for each point to each landmark inmetric space 2100. From these distances, a landmark distance (e.g.,shortest distance to a nearest landmark) for each point may beidentified and compared to generate a second set of landmark points.This process is discussed further with respect to FIGS. 23A-23D.

In step 2806, the particular computation device 2604 a may identify theshortest data point distance from among the data point distances. FIG.23A shows example landmark distances for points P₁, P₂, and P₃ tolandmark R₁ which can be used to demonstrate the selection of additionallandmark points. For example, landmark distance identification module1906 determines for each point which landmark point is the closestlandmark point for that respective point. This may include, for example,comparing the distance values d_(n) from table 2250 for each point todetermine which distance d_(n) is the shortest. Accordingly, in thisexample, the shortest between a landmark and P₁ is 3 (i.e., between P₁and landmark point R₁) and the shortest distance to a landmark pointfrom P₂ is 5 which is also to landmark point R₁.

Such an operation may use an indexable state for X (i.e., points such asP₁, P₂, and P₃ in metric space 2100), an indexable array for L (e.g.,L[l] is the index in X of the l'th landmark) where each random landmarkpoint R_(n) and subsequently determined landmark point is in L, anddClosest[x] which records the shortest distance between X[x] (i.e., P₁,P₂, P₃, etc.) and a respective closest landmark point, and inL[ ] withis true if x is in L.

In step 2808, the particular computation device 2604 a stores theshortest distance from each non-landmark point to a landmark point (orthe distance to the nearest landmark) in an array. FIG. 23B showsexample shortest distances from each non-landmark point to each landmarkpoint. In FIG. 23B, table 2350 contains the shortest distances betweeneach data point P₁, P₂, and P₃ and its closest landmark, respectively.

In various embodiments, for each non-landmark point, the closestlandmark point is identified. As a result, a list of non-landmark pointsthat identify the same landmark point as the closest landmark point maybe identified. For example, for each such landmark point, a table suchas table 2350 may be generated that identifies the non-landmark pointsthat identify the same particular landmark point as being closest. Thetable 2350 may further identify distances between those non-landmarkpoints and the same particular landmark point. In this example, table2350 may contain the shortest distances between data points P₁, P₂, andP₃ and landmark point R_(1.) data point and only one landmark R₁

In step 2810, the particular computation device 2604 a may determine alongest landmark distance from among each of the shortest data pointdistances (or a longest landmark distance) from among each of thelandmark distances. For example, returning to FIG. 23A, random landmarkpoint R₁ is the landmark nearest to points P₁, P₂, and P₃ and, thus, thelandmark distance l_(n) (i.e., the distance to a nearest landmark) foreach of these points is its respective distance to R₁, which may bestored in table 2350. Thus, in this example, the landmark distance forP₁ is l₁=3, the landmark distance for P₂ is l₂=5, and the landmarkdistance for P₃ is l₃=4. Accordingly, landmark distance comparisonmodule 1910 compares these distances to identify the longest distancewhich, in this example, is l₂=5 shown circled in FIG. 23B, belonging topoint P₂.

Thus, with the longest landmark distance, P₂ is maximally far away fromthe random landmarks relative to the other non-landmark points and, atstep 2812, the particular computation device 2604 a adds P₂ to the setof random landmark points (or seed landmarks) to generate a new set oflandmark points (e.g., the data point is added to the first sub-subsetof landmarks to expand the first sub-subset of landmarks to generate anexpanded set of landmarks). Thus, there is an initial set of randomlyselected landmark points (R) and max-min landmark points (MM) calculatedalong the way are subsequently added to R to generate a set of landmarks(L) (e.g., the expanded landmark subset of landmark points).Accordingly, FIG. 23C shows point P₂ as new MM landmark point L₁. Theaddition of the new data point to the initial subset of landmark pointscreates an expanded landmark subset of landmark points.

In step 2814, the computation device 2604 a may add other data points tothe expanded landmark subset of landmark points (e.g., the firstsub-subset of landmarks) in a similar fashion until a maximum size(e.g., a predetermined number of members) of the expanded landmarksubset of landmark points is reached. For example, i variousembodiments, this process may start over to identify and add a secondmost maximally far away point to the set of landmark points after L₁ hasbeen added to the initial set of randomly selected landmark points (R).Thus, these steps can be repeated with L₁ included into the set oflandmark points (L) when determining the landmark distances for eachpoint. Accordingly, FIG. 23D shows subset 2102 with L₁ as a new landmarkwhere the distances between various points have been calculated. In thisexample, R₁ is no longer the closest landmark to points P₁ and P₃ withthe inclusion of L₁ and L₂. For example, P₁ is now a distance d_(1′)=2from its nearest landmark L₁ and P₃, whose nearest landmark is also L₁,is now a distance d_(3′)=2 from L₁. Further, as shown in FIG. 23D, thedistance d_(4′)=3 between point P₄ and R₁ and the distance d_(5′)=4between point P₄ and newly added MM landmark point L₂ since d_(5′) islarger than d_(4′), d_(3′), and d_(1′).

It will be appreciated that the expanded landmark subset of landmarkpoints may approximate the behavior of the larger data subset therebyallowing analysis of and/or using the expanded landmark subset oflandmark points to overcome limitations, for computational efficiency,and/or speed.

FIG. 29 is a flowchart for a method of the analysis system 2606 toidentify nodes (e.g., graphical nodes or vertices) associated with oneor more of the landmark points of the analysis landmark set of landmarkpoints. FIG. 29 is an example of the analysis system 2606 applying TDAto identify nodes (e.g., graphical nodes which may be vertices). Theidentification of nodes does not necessarily require generation of avisualization (e.g., a visual representation of a graph of nodes andedges).

Computation efficiency is gained by applying TDA to identify nodesassociated with landmark points of the analysis landmark set of landmarkpoints rather than the entire large data set. Non-landmark points may besubsequently added to one or more nodes without applying TDA analysis(e.g., shown in FIG. 29) to each non-landmark point. This process ofadding non-landmark points to nodes (e.g., graphical nodes or vertices)is further discussed with regard to FIGS. 30-32.

This process of determining nodes is further described herein (e.g.,with regard to FIGS. 4, 8, and 12). In step 2902, the analysis system2606 receives a similarity function. The similarity function may beprovided by a user and/or a digital device. In some embodiments, thesimilarity function is predetermined. The similarity function is afunction that may provide a similarity measure to identify similaritybetween data points. In this example, the analysis system 2606 mayreceive a density function selection (e.g., a density estimationfunction selection). It will be appreciated that this process may becharacterized as a filter function. Examples include, but are notlimited to a Gaussian distribution.

In step 2904, the analysis system 2606 executes the selected filter(s)on the analysis landmark set of landmark points to map those landmarkpoints into a reference space. In one example, a density estimationfunction, which is well known in the art, may be performed on theanalysis landmark set of landmark points.

In step 2906, the analysis system 2606 may receive a resolutionselection. The analysis system 2606 may apply the resolution selectionto identify overlapping portions of the reference space (e.g., a coverof the reference space R) in step 2908. The application of theresolution selection generates a cover of the reference space. Asdiscussed herein, the cover of R may be a finite collection of open sets(in the metric of R) such that every point in R lies in at least one ofthese sets. In various examples, R is k-dimensional Euclidean space,where k is the number of filter functions. It will be appreciated thatthe cover of the reference space R may be controlled by the number ofintervals and the overlap identified in the resolution (e.g., see FIG.7). For example, the more intervals, the finer the resolution in S(e.g., the similarity space of the received biological data)—that is,the fewer landscape points in each S(d), but the more similar (withrespect to the filters) these points may be. The greater the overlap,the more times that clusters in S(d) may intersect clusters in S(e)—thismeans that more “relationships” between landscape points may appear,but, in some embodiments, the greater the overlap, the more likely thataccidental relationships may appear.

In step 2910, the analysis system 2606 receives a metric to cluster theinformation of the cover in the reference space to partition S(d). Inone example, the metric may be a Pearson Correlation. The clusters mayform the groupings (e.g., nodes or balls). Various cluster means may beused including, but not limited to, a single linkage, average linkage,complete linkage, or k-means method.

As discussed herein, in some embodiments, the analysis system 2606 maynot cluster two points unless filter values are sufficiently “related”(recall that while normally related may mean “close,” the cover mayimpose a much more general relationship on the filter values, such asrelating two points s and t if ref(s) and ref(t) are sufficiently closeto the same circle in the plane where ref( ) represents one or morefilter functions). The output may be a simplicial complex, from whichone can extract its 1-skeleton. The nodes of the complex may be partialclusters, (i.e., clusters constructed from subsets of S specified as thepreimages of sets in the given covering of the reference space R).

In step 2912, the analysis system 2606 may identify nodes (e.g.,graphical nodes or vertices) using the clusters and/or “related”clusters. It will be appreciated that every landscape point of theanalysis landscape set of landscape points will be a member of at leastone node.

FIG. 30 is a flowchart for adding non-landmark points as members ofnodes in some embodiments. This process may be termed as “nodefattening” whereby each non-landmark point of the original data set ornon-landmark point of the union of landmark subsets is identified as amember of one or more nodes (e.g., graphical nodes or vertices). Addingnon-landmark points to nodes as members may be computationallyinexpensive when compared to performing analysis (e.g., TDA analysis ormost other analytical functions) on a large data set. As a result,limitations may be overcome and the analysis speed increased to generatea graph and/or a visualization of a graph of nodes and edges. Further,analysis of the underlying data using node membership may be quicker toinitiate and/or perform.

The following process with regard to FIG. 30 will be discussed usingnon-landmark points of the original large data set. It will beappreciated that the following process may utilize any data pointsincluding, for example, the non-landmark points of the union of landmarksubsets. In this example, the analysis system 2606 may generate theanalysis landmark set of landmark points by performing furtherlandmarking on the union of landmark subsets. Non-landmark points fromthe union of landmark subsets may then be added to one or more nodes asmembers in the process herein.

In step 3002, the analysis system 2606 determines distances between eachnon-landmark point and each landmark of the analysis landmark set oflandmark points. In some embodiments, the analysis system 2606 mayutilize distances received from any number of the computation devices2604 a-d and/or calculate new distances. For example, the analysissystem 2606 may determine distances between each non-landmark point ofthe original large data set and each of the landmark points of theanalysis landmark set of landmark points.

In step 3004, the analysis system 2606 identifies the closest landmarkof the analysis landmark set of landmark points using the distancesdetermined in step 3002. In various embodiments, for each non-landmarkpoint, the analysis system 2606 identifies the node (e.g., identified inFIG. 29) associated with the closest identified landmark to thatparticular non-landmark point and then adds that particular non-landmarkpoint as a member to that node. For example, for a first non-landmarkpoint, the analysis system 2606 identifies the closest landmark of theanalysis landmark set to that first non-landmark point. The analysissystem 2606 then identifies the node of which the closest landmark is amember and then adds the first non-landmark point as a member of thatnode. This process may continue for each non-landmark point until allnon-landmark points are members of nodes.

It will be appreciated that if the distance between the non-landmarkpoint and the closest landmark is sufficiently long, then thenon-landmark point may be added as a member of a new node (i.e., a nodewith a membership of that particular non-landmark point) rather than amember of a previously existing node (i.e., a node with the closestlandmark point as being at least one member).

For example, in some embodiments, for each non-landmark point, theanalysis system 2606 may compare the distance between that particularnon-landmark point and its closest landmark point to a node threshold instep 3006. A node threshold is a predetermined distance. In step 3008,if the distance between a particular non-landmark point and its closestlandmark point is less than (or equal to in some embodiments) the nodethreshold, the analysis system 2606 may add the particular non-landmarkpoint as a member of the node of that closest landmark. Alternately, instep 3010, if the distance between that particular non-landmark pointand its closest landmark point is greater than (or equal to in someembodiments) the node threshold, the analysis system 2606 may create anew node for the non-landmark point and make the non-landmark point amember of the new node.

In some embodiments, the new node may be placed on the graph relative toother nodes based on the distance between the particular non-landmarkpoint and the closest landmark point.

FIG. 31 is another flowchart for adding non-landmark points as membersof nodes in some embodiments. This process is similar to that discussedwith regard to FIG. 30, however, membership of the non-landmark node maybe based on a group of closest landmark points rather than only onelandmark point. As discussed with regard to FIG. 30, the followingprocess will be discussed using non-landmark points of the originallarge data set. It will be appreciated that the following process mayutilize any data points including, for example, the non-landmark pointsof the union of landmark subsets. In this example, the analysis system2606 may generate the analysis landmark set of landmark points byperforming further landmarking on the union of landmark subsets.Non-landmark points from the union of landmark subsets may then be addedto one or more nodes as members in the process herein.

In step 3102, the analysis system 2606 determines distances between eachnon-landmark point and each landmark of the analysis landmark set oflandmark points. In some embodiments, the analysis system 2606 mayutilize distances received from any number of the computation devices2604 a-d and/or calculate new distances. For example, the analysissystem 2606 may determine distances between each non-landmark point ofthe original large data set and each of the landmark points of theanalysis landmark set of landmark points.

In step 3104, the analysis system 2606 identifies k closest landmarks ofthe analysis landmark set of landmark points using the distancesdetermined. K is a predetermined integer greater than 1. In someembodiments, a user or a digital device may provide the value of k. Kcan be any integer.

In various embodiments, for each non-landmark point, the analysis system2606 identifies the nodes (e.g., identified in FIG. 29) associated withthe k closest identified landmark points to that particular non-landmarkpoint in step 3106. In step 3108, for each non-landmark point, theanalysis system 2606 may determine which node includes the majority ofthe k closest identified landmark points and then, in step 3110, theanalysis system 2606 may assign that particular non-landmark point tothe node. If no node has a majority of the closest identified landmarkpoints as members, the analysis system 2606 may assign the non-landmarkpoint as a member of the node with the closest non-landmark point.

It will be appreciated that the analysis system 2606 may assignmembership of any particular non-landmark point in any number of ways.For example, the analysis system 2606 may order the k closest identifiedlandmark points based on distance to the particular non-landmark point.If no node has a majority of the closest identified landmark points asmembers, the analysis system 2606 may assign membership of thenon-landmark point to the node containing the closest two landmarkpoints of the k closest identified landmark points.

In various embodiments, the analysis system 2606 may assign membershipof the non-landmark point to the node containing the closest m landmarkpoints of the k closest identified landmark points (m being apredetermined integer less than k). The value of m may be provided by adigital device and/or a user. In some embodiments, if there are no nodescontaining the closest m landmark points of the k closest identifiedlandmark points, the analysis system 2606 may assign membership of thenon-landmark point to the node containing the closest m−1 landmarkpoints of the k closest identified landmark points. This process maycontinue if there are no nodes closest to m−1 landmark points. Forexample, if there are no nodes containing the closest m−1 landmarkpoints of the k closest identified landmark points, the analysis system2606 may assign membership of the non-landmark point to the nodecontaining the closest m−2 landmark points of the k closest identifiedlandmark points and so on.

FIG. 32 is a flowchart for adding non-landmark points as members ofnodes that share one or more characteristic(s) in some embodiments. Thisprocess is similar to that discussed with regard to FIG. 31, however,membership of the non-landmark node may be based on a group of closestlandmark points that share characteristic(s). The following process willbe discussed using non-landmark points of the original large data set.It will be appreciated that the following process may utilize any datapoints including, for example, the non-landmark points of the union oflandmark subsets. In this example, the analysis system 2606 may generatethe analysis landmark set of landmark points by performing furtherlandmarking on the union of landmark subsets. Non-landmark points fromthe union of landmark subsets may then be added to one or more nodes asmembers in the process herein.

In step 3102, for each non-landmark data point, the analysis system 2606may identify one or more characteristics. The characteristics mayinformation regarding the data point from the original data set. Forexample, the characteristic(s) may include values associated with one ormore columns of the data set. The analysis system 2606 may thendetermine distances between each non-landmark point and each landmark ofthe analysis landmark set that shares the characteristic(s) with thenon-landmark point. This approach may be called “neighborhood lensing.”

It will be appreciated that the analysis system 2606 may determine if anon-landmark point shares characteristic(s) with one or more landmarkpoints in any number of ways. In some embodiments, the analysis system2606 may apply a function to one or more characteristic(s) of aparticular non-landmark point and the landmark points. For example, theanalysis system 2606 may determine if a non-landmark point indicatesthat a particular test was conducted at or before a certain date (e.g.,the data points of the data sets may represent patients and at leastsome of the characteristics of the patients may indicate if a test wasperformed and when). The analysis system 2606 may determine the landmarkpoints that also indicate that the same test was conducted at or beforea certain date (e.g., which landmark points share the characteristic(s)of the non-landmark point). The analysis system 2606 may then determinemembership of the non-landmark point as discussed herein using only thelandmark points that share characteristics.

This process may be performed in any way. For example, the analysissystem 2606 may determine membership of a landmark point using only thelandmark points that do not share the characteristics with thenon-landmark point, share a degree of similarity between any number ofcharacteristics, or share a degree of dissimilarity between any numberof characteristics.

In step 3204, the analysis system 2606 determines distances between eachnon-landmark point and each landmark of the analysis landmark set oflandmark points that share the characteristic(s) with the respectivenon-landmark point. In some embodiments, the analysis system 2606 mayutilize distances received from any number of the computation devices2604 a-d and/or calculate new distances. For example, the analysissystem 2606 may determine distances between each non-landmark point ofthe original large data set and each of the landmark points of theanalysis landmark set of landmark points share characteristic(s) withthe respective non-landmark point.

In step 3206, the analysis system 2606 identifies k closest landmarks ofthe analysis landmark set of landmark points that sharecharacteristic(s) with the respective non-landmark point using thedistances determined. K is a predetermined integer greater than 1. Insome embodiments, a user or a digital device may provide the value of k.K can be any integer.

In various embodiments, for each non-landmark point, the analysis system2606 identifies the nodes (e.g., identified in FIG. 29) associated withthe k closest identified landmark points share characteristic(s) withthe particular non-landmark point in step 3208. In step 3210, for eachnon-landmark point, the analysis system 2606 may determine which nodeincludes the majority of the k closest identified landmark points andthen, in step 3212, the analysis system 2606 may assign that particularnon-landmark point to the node. As discussed herein, if no node has amajority of the closest identified landmark points as members, theanalysis system 2606 may assign the non-landmark point as a member ofthe node with the closest non-landmark point that share thecharacertistic(s).

It will be appreciated that the analysis system 2606 may assignmembership of any particular non-landmark point in any number of ways.For example, the analysis system 2606 may order the k closest identifiedlandmark points based on distance to the particular non-landmark point.If no node has a majority of the closest identified landmark points asmembers, the analysis system 2606 may assign membership of thenon-landmark point to the node containing the closest two landmarkpoints of the k closest identified landmark points.

In various embodiments, the analysis system 2606 may assign membershipof the non-landmark point to the node containing the closest m landmarkpoints of the k closest identified landmark points (m being apredetermined integer less than k). The value of m may be provided by adigital device and/or a user. In some embodiments, if there are no nodescontaining the closest m landmark points of the k closest identifiedlandmark points, the analysis system 2606 may assign membership of thenon-landmark point to the node containing the closest m−1 landmarkpoints of the k closest identified landmark points. This process maycontinue. For example, if there are no nodes containing the closest m−1landmark points of the k closest identified landmark points, theanalysis system 2606 may assign membership of the non-landmark point tothe node containing the closest m−2 landmark points of the k closestidentified landmark points and so on.

The above-described functions and components can be comprised ofinstructions that are stored on a storage medium (e.g., a computerreadable storage medium). The instructions can be retrieved and executedby a processor. Some examples of instructions are software, programcode, and firmware. Some examples of storage medium are memory devices,tape, disks, integrated circuits, and servers. The instructions areoperational when executed by the processor (e.g., a data processingdevice) to direct the processor to operate in accord with embodiments ofthe present invention. Those skilled in the art are familiar withinstructions, processor(s), and storage medium.

The present invention has been described above with reference toexemplary embodiments. It will be apparent to those skilled in the artthat various modifications may be made and other embodiments can be usedwithout departing from the broader scope of the invention. Therefore,these and other variations upon the exemplary embodiments are intendedto be covered by the present invention.

What is claimed is:
 1. A method comprising: receiving a large number ofdata points; determining at least one size of a plurality of subsets ofthe large number of data points based on constraints of at least one ofa plurality of computation devices or an analysis server, each datapoint of the large number of data points being a member of at least oneof the plurality of subsets of the large number of data points;transferring each of the plurality of subsets of large number of datapoints to a respective one of the plurality of computation devices; foreach of the plurality of subsets of data points by an associatedcomputation device of the plurality of computation devices: selecting,by the associated computation device, a group of data points from thesubset of data points to generate a first sub-subset of landmarks;adding, by the associated computation device, a non-landmark data pointof the subset of data points to the first sub-subset of landmarks tocreate an expanded sub-subset of landmarks, adding the non-landmark datapoints comprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; andconnecting at least two of the plurality of nodes with an edge if the atleast two of the plurality of nodes share at least one landmark point ofthe analysis landmark set as a member, for each data point that is botha member of the large data set but is not a member of the analysislandmark set: determining a distance between that data point and alllandmark points of the analysis landmark set; identifying a closestlandmark of the analysis landmark set to that data point; identifyingnode that includes the closest landmark of the analysis landmark set;and adding that data point as a member of the node that includes theclosest landmark of the analysis landmark set.
 2. The method of claim 1further comprising generating a visualization of the plurality of nodesand edges.
 3. A method comprising: receiving a large number of datapoints; determining at least one size of a plurality of subsets of thelarge number of data points based on constraints of at least one of aplurality of computation devices or an analysis server, each data pointof the large number of data points being a member of at least one of theplurality of subsets of the large number of data points; transferringeach of the plurality of subsets of large number of data points to arespective one of the plurality of computation devices; for each of theplurality of subsets of data points by an associated computation deviceof the plurality of computation devices: selecting, by the associatedcomputation device, a group of data points from the subset of datapoints to generate a first sub-subset of landmarks; adding, by theassociated computation device, a non-landmark data point of the subsetof data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, adding the non-landmark data pointscomprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; andconnecting at least two of the plurality of nodes with an edge if the atleast two of the plurality of nodes share at least one landmark point ofthe analysis landmark set as a member, for each data point that is botha member of the large data set but is not a member of the analysislandmark set: determining a distance between that data point and alllandmark points of the analysis landmark set; identifying a closestlandmark of the analysis landmark set to that data point; comparing adistance between the closest landmark of the analysis landmark set andthat data point to a node threshold; and if the distance between theclosest landmark of the analysis landmark set and that data point isgreater than the node threshold, generating a new node including thatdata point as a member of the new node; if the distance the distancebetween the closest landmark of the analysis landmark set and that datapoint is less than the node threshold, adding that data point as amember of the node that includes the closest landmark of the analysislandmark set.
 4. A method comprising: receiving a large number of datapoints; determining at least one size of a plurality of subsets of thelarge number of data points based on constraints of at least one of aplurality of computation devices or an analysis server, each data pointof the large number of data points being a member of at least one of theplurality of subsets of the large number of data points; transferringeach of the plurality of subsets of large number of data points to arespective one of the plurality of computation devices; for each of theplurality of subsets of data points by an associated computation deviceof the plurality of computation devices: selecting, by the associatedcomputation device, a group of data points from the subset of datapoints to generate a first sub-subset of landmarks; adding, by theassociated computation device, a non-landmark data point of the subsetof data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, adding the non-landmark data pointscomprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; andconnecting at least two of the plurality of nodes with an edge if the atleast two of the plurality of nodes share at least one landmark point ofthe analysis landmark set as a member, for each data point that is botha member of the large data set but is not a member of the analysislandmark set: determining a distance between that data point and alllandmark points of the analysis landmark set; identifying apredetermined number of closest landmark of the analysis landmark set tothat data point; identifying a node which includes a majority of thepredetermined number of closest landmarks of the analysis landmark setas members; and adding that data point as a member of the node thatincludes a majority of the predetermined number of closest landmarks ofthe analysis landmark set as members.
 5. A method comprising: receivinga large number of data points; determining at least one size of aplurality of subsets of the large number of data points based onconstraints of at least one of a plurality of computation devices or ananalysis server, each data point of the large number of data pointsbeing a member of at least one of the plurality of subsets of the largenumber of data points; transferring each of the plurality of subsets oflarge number of data points to a respective one of the plurality ofcomputation devices; for each of the plurality of subsets of data pointsby an associated computation device of the plurality of computationdevices: selecting, by the associated computation device, a group ofdata points from the subset of data points to generate a firstsub-subset of landmarks; adding, by the associated computation device, anon-landmark data point of the subset of data points to the firstsub-subset of landmarks to create an expanded sub-subset of landmarks,adding the non-landmark data points comprising: calculating first datapoint distances between each non-landmark data point and each landmark;identifying a shortest data point distance from among the first datapoint distances for each non-landmark data point; identifying aparticular non-landmark data point with a longest first landmarkdistance of all the shortest data point distances; and adding theparticular non-landmark data point to the first sub-subset of landmarksto expand the first sub-subset of landmarks to generate an expanded setof landmarks until the expanded sub-subset of the expanded landmarksreaches a predetermined number of members, repeat adding thenon-landmark data points; creating an analysis landmark set based on acombination of expanded sub-subsets of expanded landmarks; performing asimilarity function on the analysis landmark set to map landmark pointsof the analysis landmark set to a mathematical reference space;generating a cover of the mathematical reference space to divide themathematical reference space into overlapping subsets; clustering themapped landmark points of the analysis landmark set based on theoverlapping subsets of the cover in the mathematical reference space;creating a plurality of nodes, each of the plurality of nodes beingbased on the clustering of the mapped landmark points of the analysislandmark set, each landmark point of the analysis landmark set being amember of at least one node; connecting at least two of the plurality ofnodes with an edge if the at least two of the plurality of nodes shareat least one landmark point of the analysis landmark set as a member;and generating a visualization of the plurality of nodes and edges. 6.The method of claim 5, wherein selecting, by the associated computationdevice, the group of data points from the subset of data points togenerate the first sub-subset of landmarks is performed randomly.
 7. Amethod comprising: receiving a large number of data points; determiningat least one size of a plurality of subsets of the large number of datapoints based on constraints of at least one of a plurality ofcomputation devices or an analysis server, each data point of the largenumber of data points being a member of at least one of the plurality ofsubsets of the large number of data points; transferring each of theplurality of subsets of large number of data points to a respective oneof the plurality of computation devices; for each of the plurality ofsubsets of data points by an associated computation device of theplurality of computation devices: selecting, by the associatedcomputation device, a group of data points from the subset of datapoints to generate a first sub-subset of landmarks; adding, by theassociated computation device, a non-landmark data point of the subsetof data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, adding the non-landmark data pointscomprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; connecting atleast two of the plurality of nodes with an edge if the at least two ofthe plurality of nodes share at least one landmark point of the analysislandmark set as a member; and determining the predetermined number ofmembers of the expanded sub-subset of the expanded landmarks based onthe constraints of the at least one of a plurality of computationdevices or an analysis server.
 8. The method of claim 7, wherein thedetermination of the predetermined number of members of the expandedsub-subset of the expanded landmarks is based, at least in part, on adetermination of a predetermined number of members of the analysislandmark set.
 9. A non-transitory computer readable medium comprisinginstructions executable by a processor to perform a method, the methodcomprising: receiving a large number of data points; transferring eachof the plurality of subsets of large number of data points to arespective one of the plurality of computation devices, each of anassociated computation device of the plurality of computation devices:selecting, by the associated computation device, a group of data pointsfrom the subset of data points to generate a first sub-subset oflandmarks; adding, by the associated computation device, a non-landmarkdata point of the subset of data points to the first sub-subset oflandmarks to create an expanded sub-subset of landmarks, adding thenon-landmark data points comprising: calculating first data pointdistances between each non-landmark data point and each landmark;identifying a shortest data point distance from among the first datapoint distances for each non-landmark data point; identifying aparticular non-landmark data point with a longest first landmarkdistance of all the shortest data point distances; and adding theparticular non-landmark data point to the first sub-subset of landmarksto expand the first sub-subset of landmarks to generate an expanded setof landmarks until the expanded sub-subset of the expanded landmarksreaches a predetermined number of members, repeat adding thenon-landmark data points; creating an analysis landmark set based on acombination of expanded sub-subsets of expanded landmarks; performing asimilarity function on the analysis landmark set to map landmark pointsof the analysis landmark set to a mathematical reference space;generating a cover of the mathematical reference space to divide themathematical reference space into overlapping subsets; clustering themapped landmark points of the analysis landmark set based on theoverlapping subsets of the cover in the mathematical reference space;creating a plurality of nodes, each of the plurality of nodes beingbased on the clustering of the mapped landmark points of the analysislandmark set, each landmark point of the analysis landmark set being amember of at least one node; and connecting at least two of theplurality of nodes with an edge if the at least two of the plurality ofnodes share at least one landmark point of the analysis landmark set asa member, for each data point that is both a member of the large dataset but is not a member of the analysis landmark set: determining adistance between that data point and all landmark points of the analysislandmark set; identifying a closest landmark of the analysis landmarkset to that data point; identifying node that includes the closestlandmark of the analysis landmark set; and adding that data point as amember of the node that includes the closest landmark of the analysislandmark set.
 10. The non-transitory computer readable medium of claim 9further comprising generating a visualization of the plurality of nodesand edge.
 11. A non-transitory computer readable medium comprisinginstructions executable by a processor to perform a method, the methodcomprising: receiving a large number of data points; transferring eachof the plurality of subsets of large number of data points to arespective one of the plurality of computation devices, each of anassociated computation device of the plurality of computation devices:selecting, by the associated computation device, a group of data pointsfrom the subset of data points to generate a first sub-subset oflandmarks; adding, by the associated computation device, a non-landmarkdata point of the subset of data points to the first sub-subset oflandmarks to create an expanded sub-subset of landmarks, adding thenon-landmark data points comprising: calculating first data pointdistances between each non-landmark data point and each landmark;identifying a shortest data point distance from among the first datapoint distances for each non-landmark data point; identifying aparticular non-landmark data point with a longest first landmarkdistance of all the shortest data point distances; and adding theparticular non-landmark data point to the first sub-subset of landmarksto expand the first sub-subset of landmarks to generate an expanded setof landmarks until the expanded sub-subset of the expanded landmarksreaches a predetermined number of members, repeat adding thenon-landmark data points; creating an analysis landmark set based on acombination of expanded sub-subsets of expanded landmarks; performing asimilarity function on the analysis landmark set to map landmark pointsof the analysis landmark set to a mathematical reference space;generating a cover of the mathematical reference space to divide themathematical reference space into overlapping subsets; clustering themapped landmark points of the analysis landmark set based on theoverlapping subsets of the cover in the mathematical reference space;creating a plurality of nodes, each of the plurality of nodes beingbased on the clustering of the mapped landmark points of the analysislandmark set, each landmark point of the analysis landmark set being amember of at least one node; and connecting at least two of theplurality of nodes with an edge if the at least two of the plurality ofnodes share at least one landmark point of the analysis landmark set asa member, for each data point that is both a member of the large dataset but is not a member of the analysis landmark set: determining adistance between that data point and all landmark points of the analysislandmark set; identifying a closest landmark of the analysis landmarkset to that data point; comparing a distance between the closestlandmark of the analysis landmark set and that data point to a nodethreshold; and if the distance between the closest landmark of theanalysis landmark set and that data point is greater than the nodethreshold, generating a new node including that data point as a memberof the new node; if the distance the distance between the closestlandmark of the analysis landmark set and that data point is less thanthe node threshold, adding that data point as a member of the node thatincludes the closest landmark of the analysis landmark set.
 12. Anon-transitory computer readable medium comprising instructionsexecutable by a processor to perform a method, the method comprising:receiving a large number of data points; transferring each of theplurality of subsets of large number of data points to a respective oneof the plurality of computation devices, each of an associatedcomputation device of the plurality of computation devices: selecting,by the associated computation device, a group of data points from thesubset of data points to generate a first sub-subset of landmarks;adding, by the associated computation device, a non-landmark data pointof the subset of data points to the first sub-subset of landmarks tocreate an expanded sub-subset of landmarks, adding the non-landmark datapoints comprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; andconnecting at least two of the plurality of nodes with an edge if the atleast two of the plurality of nodes share at least one landmark point ofthe analysis landmark set as a member, for each data point that is botha member of the large data set but is not a member of the analysislandmark set: determining a distance between that data point and alllandmark points of the analysis landmark set; identifying apredetermined number of closest landmark of the analysis landmark set tothat data point; identifying a node which includes a majority of thepredetermined number of closest landmarks of the analysis landmark setas members; and adding that data point as a member of the node thatincludes a majority of the predetermined number of closest landmarks ofthe analysis landmark set as members.
 13. A non-transitory computerreadable medium comprising instructions executable by a processor toperform a method, the method comprising: receiving a large number ofdata points; transferring each of the plurality of subsets of largenumber of data points to a respective one of the plurality ofcomputation devices, each of an associated computation device of theplurality of computation devices: selecting, by the associatedcomputation device, a group of data points from the subset of datapoints to generate a first sub-subset of landmarks; adding, by theassociated computation device, a non-landmark data point of the subsetof data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, adding the non-landmark data pointscomprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; connecting atleast two of the plurality of nodes with an edge if the at least two ofthe plurality of nodes share at least one landmark point of the analysislandmark set as a member; and generating a visualization of theplurality of nodes and edges.
 14. The non-transitory computer readablemedium of claim 13, wherein selecting, by the associated computationdevice, the group of data points from the subset of data points togenerate the first sub-subset of landmarks is performed randomly.
 15. Anon-transitory computer readable medium comprising instructionsexecutable by a processor to perform a method, the method comprising:receiving a large number of data points; transferring each of theplurality of subsets of large number of data points to a respective oneof the plurality of computation devices, each of an associatedcomputation device of the plurality of computation devices: selecting,by the associated computation device, a group of data points from thesubset of data points to generate a first sub-subset of landmarks;adding, by the associated computation device, a non-landmark data pointof the subset of data points to the first sub-subset of landmarks tocreate an expanded sub-subset of landmarks, adding the non-landmark datapoints comprising: calculating first data point distances between eachnon-landmark data point and each landmark; identifying a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identifying a particular non-landmark datapoint with a longest first landmark distance of all the shortest datapoint distances; and adding the particular non-landmark data point tothe first sub-subset of landmarks to expand the first sub-subset oflandmarks to generate an expanded set of landmarks until the expandedsub-subset of the expanded landmarks reaches a predetermined number ofmembers, repeat adding the non-landmark data points; creating ananalysis landmark set based on a combination of expanded sub-subsets ofexpanded landmarks; performing a similarity function on the analysislandmark set to map landmark points of the analysis landmark set to amathematical reference space; generating a cover of the mathematicalreference space to divide the mathematical reference space intooverlapping subsets; clustering the mapped landmark points of theanalysis landmark set based on the overlapping subsets of the cover inthe mathematical reference space; creating a plurality of nodes, each ofthe plurality of nodes being based on the clustering of the mappedlandmark points of the analysis landmark set, each landmark point of theanalysis landmark set being a member of at least one node; connecting atleast two of the plurality of nodes with an edge if the at least two ofthe plurality of nodes share at least one landmark point of the analysislandmark set as a member; and determining the predetermined number ofmembers of the expanded sub-subset of the expanded landmarks based onthe constraints of the at least one of a plurality of computationdevices or an analysis server.
 16. The non-transitory computer readablemedium of claim 15, wherein the determination of the predeterminednumber of members of the expanded sub-subset of the expanded landmarksis based, at least in part, on a determination of a predetermined numberof members of the analysis landmark set.
 17. A system comprising: atleast one processor; and memory configured to contain instructions tocontrol the processor to: receive a large number of data points;determine at least one size of a plurality of subsets of the largenumber of data points based on constraints of at least one of aplurality of computation devices or an analysis server, each data pointof the large number of data points being a member of at least one of theplurality of subsets of the large number of data points; transfer eachof the plurality of subsets of large number of data points to arespective one of the plurality of computation devices to enable foreach of the plurality of subsets of data points by an associatedcomputation device of the plurality of computation devices to: select,by the associated computation device, a group of data points from thesubset of data points to generate a first sub-subset of landmarks; add,by the associated computation device, a non-landmark data point of thesubset of data points to the first sub-subset of landmarks to create anexpanded sub-subset of landmarks, adding the non-landmark data pointscomprising: calculate first data point distances between eachnon-landmark data point and each landmark; identify a shortest datapoint distance from among the first data point distances for eachnon-landmark data point; identify a particular non-landmark data pointwith a longest first landmark distance of all the shortest data pointdistances; and add the particular non-landmark data point to the firstsub-subset of landmarks to expand the first sub-subset of landmarks togenerate an expanded set of landmarks until the expanded sub-subset ofthe expanded landmarks reaches a predetermined number of members, repeatadding the non-landmark data points; create an analysis landmark setbased on a combination of expanded sub-subsets of expanded landmarks;perform a similarity function on the analysis landmark set to maplandmark points of the analysis landmark set to a mathematical referencespace; generate a cover of the mathematical reference space to dividethe mathematical reference space into overlapping subsets; cluster themapped landmark points of the analysis landmark set based on theoverlapping subsets of the cover in the mathematical reference space;create a plurality of nodes, each of the plurality of nodes being basedon the clustering of the mapped landmark points of the analysis landmarkset, each landmark point of the analysis landmark set being a member ofat least one node; connect at least two of the plurality of nodes withan edge if the at least two of the plurality of nodes share at least onelandmark point of the analysis landmark set as a member; and generatinga visualization of the plurality of nodes and edges.